Paul Jones was so gracious as to mention my name in a presentation he gave in Atlanta, a couple of my coworkers where there but I wasn't able to make it, instead I am going to the ZendCon.
Anyway since it was about benchmarking it made me think back to my orignal tests where the Zend Framework came out around 18% the performance of a plain html and Paul claimed Solar was 4 times faster then ZF.
Why test against plain html or plain php? To get a baseline of the fastest possible performance you can get. It can give you a reasonable goal, for example if a plain page gets 100rps and a php page gets 50rps you know there is room up to 50rps.. What you have to do is decide what your reasonable goal is. Its sort of like weight loss, If you set unreasonable goals your going to lose. If you set a goal of 10% or 5rps improvement thats a goal that you can reach.
So Zend Framework is now at 1.0.1, what did benchmarking find? The goal is to just output Hello World with the most basic set of files possible for each case.
ab -c 10 -t 60 http://zend.cyberlot.net/
Plain HTML
------------------------------------------
Concurrency Level: 10
Time taken for tests: 14.553157 seconds
Complete requests: 50000
Requests per second: 3435.68 [#/sec] (mean)
Time per request: 2.911 [ms] (mean)
Time per request: 0.291 [ms] (mean, across all concurrent requests)
Transfer rate: 915.95 [Kbytes/sec] received
Pure PHP
------------------------------------------
Concurrency Level: 10
Time taken for tests: 17.330895 seconds
Complete requests: 50000
Requests per second: 2885.02 [#/sec] (mean)
Time per request: 3.466 [ms] (mean)
Time per request: 0.347 [ms] (mean, across all concurrent requests)
Transfer rate: 571.93 [Kbytes/sec] received
Zend Framework 1.0.1
-------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.13438 seconds
Complete requests: 3633
Requests per second: 60.54 [#/sec] (mean)
Time per request: 165.190 [ms] (mean)
Time per request: 16.519 [ms] (mean, across all concurrent requests)
Transfer rate: 12.00 [Kbytes/sec] received
That sucked pretty bad had to figure out why, I know they have added a lot but not that much. Well what about our original tests.
Zend Framework 0.2.0
----------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.39417 seconds
Complete requests: 6420
Requests per second: 106.93 [#/sec] (mean)
Time per request: 93.519 [ms] (mean)
Time per request: 9.352 [ms] (mean, across all concurrent requests)
Transfer rate: 21.20 [Kbytes/sec] received
Better but still pretty poor compared to my old results, the basic php and html results are faster then before, framework is slower?
APC is whats different, a recent recompile left my apc disabled, Fixed this and reran
Zend Framework 1.0.1
------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.7181 seconds
Complete requests: 13347
Requests per second: 222.42 [#/sec] (mean)
Time per request: 44.959 [ms] (mean)
Time per request: 4.496 [ms] (mean, across all concurrent requests)
Transfer rate: 44.09 [Kbytes/sec] received
Zend Framework 0.2.0
-------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.119 seconds
Complete requests: 22109
Requests per second: 368.48 [#/sec] (mean)
Time per request: 27.138 [ms] (mean)
Time per request: 2.714 [ms] (mean, across all concurrent requests)
Transfer rate: 73.05 [Kbytes/sec] received
What does this tell you? If your not running apc, zend or another package your shooting yourself in the foot.
It also shows a nice a pretty unhealthy drop between 0.2.0 and 1.0.1 for something as simple as "Hello World". While it took me fewer lines and is easier to setup is it worth it?
A nod to Paul
Solar 0.28
--------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.080 seconds
Complete requests: 18841
Requests per second: 314.02 [#/sec] (mean)
Time per request: 31.845 [ms] (mean)
Time per request: 3.185 [ms] (mean, across all concurrent requests)
Transfer rate: 122.35 [Kbytes/sec] received
Its late and I didn't have time to customize Solar, It has a "hello-mini" built in so I used that and all the default settings based on Getting Started. Hats off to Paul, back in the 0.2.0 days the whole reason I ran my benchmarks is I couldn't understand why his results where so slow and found ZF to be just as fast but now Solar has a clear lead.
"Cyberlot Framework" Alpha Stripped down Zend Framework + Savant 3 (Can't help it I just like pauls work)
-----------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 34.435357 seconds
Complete requests: 50000
Requests per second: 1452.00 [#/sec] (mean)
Time per request: 6.887 [ms] (mean)
Time per request: 0.689 [ms] (mean, across all concurrent requests)
Transfer rate: 287.84 [Kbytes/sec] received
Of course my controller meets my needs and doesn't include the kitchen sink ;) But it does what I need.. just food for thought, Yes a Framework will speed up development but if specific application code will be faster.
What would I change in the Zend Framework? For development all the checks and exceptions they have are great and make debugging easier but in the real world for example my class layout and files are already decided and not moving, I don't need the Framework to check 5 different ways to make sure the file and class exists! By making these checks optional its possible to double the speed of the loader for example.
Anyway thats all for tonight, Later.
