Benchmarking Zend Framework loader

One of the things I am doing in course of my work is performance benchmarks for various stuff – PHP, Zend products, applications, etc. Performance in PHP space is currently like alchemy – there are a lot of rumors floating around about various properties of various stuff, but much less reliable data that can be verified and used. PHP has standard bench.php script, but it covers only a small part of what real-life applications do. I wish there were more established tests and methods for benchmarking PHP engine and applications. But benchmarking is a complicated subject, and variety of PHP platforms and applications makes it harder to create useful general-purpose benchmarks.

But more to the point. On Zend Framework lists there was a topic raised about performance impact of Zend_Loader component, which is used for – no surprise here! – loading classes, including autoloading, etc. Some folks thought that since Zend_Loader is executing some code before actual loading the required file, it must cost something. And it makes sense. However, how much does it cost?
Well, the best way to know the price of something is to ask – and in this case, to run the test. So that’s what I did – I made a list of 725 Framework classes (ZF now has more than 1000 but I composed the list some time ago and had also to drop some to avoid some tricky dependencies). And I wrote two scripts – one that would load these classes with require_once and one that would load them using Zend_Loader::loadClass. Both the data file and the scripts are available for download for those that would like to play with it. I tested them with and without Zend’s bytecode cache, to see how much one can save using bytecode caching technology.

So, the results were as follows:

Without bytecode cache:

          require_once Zend_Loader
php5.2        4.42      4.42
php5.3        4.96      4.97

With bytecode cache:

           require_once Zend_Loader
php5.2        63.04     56.62
php5.3        61.28     55.52

The numbers are requests per second, so more is better. Test run on Linux dual 2GHz AMD.

What we can conclude from these?

  1. It is very important to understand that it is a narrow-point benchmark that tests only one function in one specific way. Please do not draw conclusions on behavior of whole applications based only on this benchmark.
  2. You do want to use bytecode caching. You won’t get 15x performance on any real application, but it does speed up loading very significantly.
  3. Without bytecode caching, it doesn’t matter if you use require_once or Loader – both are equally slow 🙂
  4. With bytecode caching, Loader has some overhead – explanation for this is that with file accesses eliminated, require_once of course has little left, while Loader still does a couple of function calls. But on real-life apps it’d probably be very small, provided that it’s about 10% even on loading-only huge-class-list benchmark, and your application probably does something useful instead of loading 700+ framework classes :)) Meaning, fears of using the class loader vs. doing require_once are seriously overstated.
  5. 5.3 is still a moving target, to don’t put too much stake in current benchmark results for 5.3, they probably will be different by the time 5.3 is in release cycle (hopefully, better :))

P.S. This post does not talk about other things like “what if I stuff all classes I use into single file”, etc. Maybe next time.


9 thoughts on “Benchmarking Zend Framework loader

  1. Pingback: inspired

  2. Hey,
    It is great to see that research. I really wonder if we have repeated the same enough times to see the result. What I am trying to say is the that the results might be different with higher number of times.

    Keep up the research. Let us see what happens when PHP 6 comes out.

    Thanks and Regards

  3. This benchmark manages to put wrong conclusions into developer minds DESPITE clearly stating its narrow target to be just DIRECTLY comparing require_once and Zend_Loader. The biggest advantage of autoloading is completetely eliminated by the benchmark: it prevents loading any code the script doesn’t use.

    With the Zend Framework, this is even more problematic since its class files are stuffed with require_once lines just sitting there to include unused code. This is due to most of them sitting at the top of the class files, always getting included, with the script not yet knowing which methods it will be executing and which classes are truely needed. Only for some Exception classes the require_once is put directly before the “new” statement. Doing a “search/replace in files”, commenting out ALL of those require_once lines and sticking to just using the autoloader will result in less memory usage and higher overall performance.

    To further increase the performance, replace loadClass with two methods: the old one and a new one for true autoloading (regular new statement with static class name) where the expensive “preg_match” security check calls are not used and not needed, since a forbidden character in a class name right inside the php script would already have yielded a fatal error.

    Also, the most expensive operation in the autoloader when using an opcode cache is the “file_exists” call. Take it out when you are sure all classes will be found where expected. Forcing hits on the filesystem when the class has already been compiled and sits in the cache is VERY BAD for the overall performance. Sadly, taking out this call is not so easy with the usual MVC file structure in Zend Framework applications.

  4. You should sed all those underscores and make a new classes.txt for the require_once test and avoid the str_replace … save a few more milliseconds 🙂

  5. Pingback: PHP 10.0 Blog: Benchmarking Zend Framework loader | Development Blog With Code Updates :

  6. Andrew, numbers are requests per second, more means faster.

    Celebgossipbuzz, I think your information about APC is not correct. Now, I’m not APC expert (you could find those on PECL-DEV list 😉 but I see absolutely no problem with caching files with variable names. Of course, you can think of optimizations that couldn’t be applied there, but bytecode caching certainly can. Zend’s solution definitely does not have this issue, and I’d be very surprised if APC did.

    As for first time, it indeed might have take a bit longer, but not substantially so. Unfortunately, this thing is *really* hard to benchmark, since you’ve got only one first request, so most benchmarking tools and methods wouldn’t work in this case. But my guesstimate would be slowdown is negligible. That said, even if it were not, I would still think bytecode cache is worth it.

  7. I have one more question. With bytecode caching there should be a considerable difference in the execution time when you run it for the first time, and all the consecutive runs, correct ? The first one has all the class files cached, which takes more then the usual time and resources, while all the next executions will “load” the classes off the bytecode cache. Did I got this correct ?

  8. Hi, nice post. The bytecode stuff is a little blurry to me though. You have something like require_once $file; in your code — will this trigger the bytecode cache ? I mean I read about that there are issues w/ APC and that it can not bytecode cache files, which names are composed at runtime and stuffed in variables. So, if you are familiar with how the bytecode works under the hood, can you please enlighten me ? Thanks in advance.

  9. You do want to use bytecode caching.

    I think you have your performance numbers above backwards. The way they are posted, byte code caching is taking the longer period of time than the require_once method.

Comments are closed.