Using PHP APC cache in CLI mode using dumpfiles

Question

I've recently started using APC cache on our servers. One of the most important parts of our product is a CLI (Cron/scheduled) process, whose performance is critical. Typically the batchjob consists of running some 16-32 processes in parallel for about an hour (they "restart" every few minutes).

By default, using APC cache in CLI is a waste of time due to the opcode cache not being retained between individual calls. But APC also contains apc_bin_dumpfile() and apc_load_dumpfile() functions.

I was thinking these two function might be used to make APC efficient in CLI mode by having it all compiled sometime outside the batchjob, stored in a single dumpfile and having the individual processes load the dumpfile.

Does anybody have any experience with such a scenario or can you give good reasons why it will or will not work? If any significant gains could reasonably be had, either in memory use or performance? What pitfalls are lurking in the shadows?

Valencia

I didn't get the part about why is using APC a waste of time in CLI? What do you mean that by opcode not being retained between individual calls? – Darleen

APC has to rebuild the opcode cache whenever it's restarted. When used with a webserver it just stays active so long as the computer is active, but in CLI mode each process has it's own APC so for each process it has to rebuild the opcode cache. Since APC increases performance by NOT having to build the opcodes for each PHP script, it pretty much negates any performance benefit you might have had. I'm trying to figure out a way around this problem. – Valencia

Hope this doesn't come off as obtuse, but have you tried it yet? This is a clever workaround, wish I had thought about it myself. I had an issue like this before, and preferred to use Node.js (and forever) with timed loops. – Kathlyn

Does the compile time really matter in this scenario? It seems very small compared to the runtime in this case. – Sesquioxide

@wolfgang The scenario would matter mostly in memory usage. I use CLI mode to run tasks in parallel (as a poor-man's multithreading). The less memory each task requires, the more tasks I can run in parallel. – Valencia

@Rob haven't been able to try it yet. – Valencia

you'd probably win more using your code from a .phar file. – Prothesis

Answer 1

Disclaimer: As awesome as APC is when it works in CLI, and it is awesome, it can equally be as frustrating. Use with a healthy load of patience, be thorough, step away from the problem if you're spinning, keep in mind you are working with cache that is why it seems like its doing nothing, it is actually doing nothing. Delete dump file, start with just the basics, if that doesn't work forget it try a new machine, new OS, if it is working make a copy, piece by piece expand functionality - there are loads of things that won't work, if it is working commit or make a copy, add another piece and test again, for sanity-check recheck the copies that were working before, cliches or not; if at first you don't succeed try try again, you can't keep doing the same thing expecting new results.

Ready? This is what you've been waiting for:

Enable apc for cli

apc.enable-cli=1

it is not ideal to create, populate and destroy the APC cache on every CLI request

   - previous answer by unknown poster since removed.

You're absolutely right that sucks, lets fix it shall we?

If you try and use APC under CLI and it is not enabled you will get warnings.

something like:

PHP Warning:  apc_bin_loadfile(): APC is not enabled, 
              apc_bin_loadfile not available.
PHP Warning:  apc_bin_dumpfile(): APC is not enabled, 
              apc_bin_dumpfile not available.

Warning: I suggest you don't enable cli in php.ini, it is not worth the frustration, you are going to forget you did it and have numerous other headaches with other scripts, trust me its not worth it, use a launcher script instead. (see below)

apc_loadfile and apc_dumpfile in cli

As per the comment by mightye php we need to disable apc.stat or you will get a warnings

something like:

PHP Warning:  apc_bin_dumpfile(): Excluding some files from apc_bin_dump[file].  
              Cached files must be included using full path with apc.stat=0. 

launcher script - php-apc.sh

We will use this script to launch our apc enabled scripts (ex. ./php-apc.sh apc-cli.php) instead of changing the properties in php.ini directly.

#/bin/sh
php -d apc.enable_cli=1 -d apc.stat=0 $1

Ready for the basic functionality? Sure you are =)

basic APC persisted - apc-cli.php

<?php
/** check if dump file exists, you don't want to use file_exists */
if (false !== $dump_file = stream_resolve_include_path('apc.dump'))
    /** so where were we lets have a look see shall we */
    if (false !== apc_bin_loadfile($dump_file))
        /** fetch what was stored last run just for fun */
        if (false !== $value = apc_fetch('my.awesome.apc.store'))
            echo "$value from apc\n";

/** store what gets fetched the next run just for fun */
apc_store('my.awesome.apc.store', 'awesome in cli');
/** what a shlep lets not do that all over again shall we */
apc_bin_dumpfile(array(),null,'apc.dump');

Notice: Why not use file_exists? Because file_exists == stat you see and we want to reap the reward that is apc.stat=0 so; work within the include path; use absolute and not relative paths - as returned by stream_resolve_include_path(); avoid include_once, require_once use the non *_once counterparts; check your stat usage, when not using APC(Muchos important senor), with the help of a StreamWrapper echo for calls to method url_stat; Oops: Fatal scope over-run error! aborting notice thread. see url_stat message: Error caused by StreamWrapper outside the scope of this discussion.

The smoke test

Using the launcher execute the basic script

./php-apc.sh apc-cli.php

A whole bunch of nothing happened that's what we want right, why else do you want to use cache? If it did output anything then it didn't work, sorry.

There should be a dump file called apc.dump see if you can find it? If you can't find it then it didn't work, sorry.

Good we have the dump file there were no errors lets run it again.

./php-apc.sh apc-cli.php

What you want to see:

awesome in cli from apc

Success! =)

There are few in PHP as satisfying as a working APC implementation.
nJoy!

Hygrostat

On MAC OS X 10.6.8 Custom compiled PHP 5.3.14 (cli) (built: Jul 7 2012 21:32:43) – Hygrostat

Excellent answer! Sad to head about the stat limitations though. In my particular case I need to include a rather large framework riddled with includes so I may have my work cut out for me. Either way this should be a good foundation to start experimenting. – Valencia

Do you have any experience of whether enabling APC in CLI like this actually improves performance and/or memory usage? – Valencia

There aren't any limitations perse the point I was trying to make is that you want to be benefitting from the non stat optimizations with spl and apc rather than not, but everything will still work as always. Try and use include_path and stream_resolve_include_path to get absolute paths, maybe a custom autoloader would serve you too. I can probably squeeze off some time to lend a hand if you need it. – Hygrostat

If you are running out of memory it is likely that you have other problems and there is no replacement for conservation make sure you unset those memory hogs. Assigning null ie. $MyHugeCollection = null; is much quicker than unset($WhatIsTakingSoLong); but regardless you would be happier with APC if you plan not to have to redo the long operations in the next run. If you will still be doing everything then there is hardly any point in going this route. – Hygrostat

You make reference to pecl "16-32 processes" are you holding onto and cleaning up all the orphans as that will also give you lots of uphill and resource depletion? – Hygrostat

As far as I can see, most of the memory used is actually lost while the legacy code I'm including is including several more levels of code. – Valencia

I lose ~8MB memory each on two simple include_once()'s. I could trim one down to ~6MB by removing some unneeded "sub"-includes but anything more than that would require major refactoring. The processes each use a consistant ~25MB for their entire ~7 minute run (after which they die and newly started processes take over) so there doesn't seem any inefficiency in the processes itself. Most of memory seems to be wasted by simply including a lot of code. I've noticed significant memory profile improvements using this legacy code over the web with APC, so it seems the most obvious target. – Valencia

You will notice I recommend against the use of *once which stems from Luke Scott's statement that _APC does an optimization with require/include (not *_once). Have a look at get_included_files perhaps if you can't avoid re-including the same file more than once in any other way. – Hygrostat

I would not be too concerned if the script stays constant with whichever amount of memory it requires. This might be optimized but at least it can be catered for by configuring memory-limit in php.ini with enough buffer to accommodate. On the other hand if it accumulates the longer it runs then we have a leak and it will likely run out of available memory at some point, which is not ideal. – Hygrostat

Answer 2

I would definitely not use it in the CLI as when you restart it, it's almost as if it was never running in the first place!

The better way of using APC is to have it running on the webserver itself all the time, this way with it being active it will actually do what it's supposed to do!

Sneaker

Obviously, I already have it running on the webserver. But I would ALSO like to use it in CLI mode in order to reduce the memory profile. By default, APC won't help much as it has to recompile files each run, but I'd like to know if it's possible to use APC dump files to skip the compilation step. – Valencia

Yes, you can do that to skip the compilation step as long as no variables have changed in the meantime. So basically, if anything needs to be re-cached then you need to recompile! – Sneaker

Answer 3

I tryed with curl and APC.it works

use these commands in CLI

curl --data "param1=value2" http://testsite.com/test.php

so it will post data to test.php and you writes the code in it.

Rounding

Blocking your performance-critical script with an HTTP call is probably the worst thing you can do. – Olivero