PHP asynchronous processing with beanstalkd. Do you recommend it?

Question

would you recommend using beanstalkd [ http://kr.github.com/beanstalkd/ ] for

asynchronous processing in PHP ?

I need some feedback/comments pros-cons, from anyone that has used this lib.

Thanks,

Oxfordshire

I would also be interested to read comments on beanstalkd's long-term stability from someone who has used it in a production environment – Wicker

i'm very interested too. – Reed

Have you tried Gearman? I guess Gearman-PHP community is more active. – Multitude

@mixdev: Yes i have, and it proved a good solution for async processing among other things :) – Oxfordshire

Answer 1

I've used Beanstalk in production, and also while testing threw millions of simple messages through it - generally en-mass, but the production system had over a 100,000 tasks put through it till I left the company. It may still be running, in which case it would be tens of millions be now - or more, if they had extended it's use further, as I had planned.

I would recommend it, as it's got a number of excellent points.

• named tubes can be used to limit the jobs being delivered. I watch a tube based on the machine hostname, which limited where a worker would be running - useful for uploaded files which are only stored on a particular server).
• The delays can be used to set future events
• Latest versions of the server also support bin-logging, giving persistence, though it has never crashed on me.

My first task I put through it was image processing - and doing that work outside of an Apache/mod_php process allowed me to resize larger images without affecting the server (blowing out the Webserver). With a lightly loaded queue, it had created the thumbnails before the page had refreshed after the upload.

There are many other potential tasks that could also be processed asynchronously.

The only problems I have ever had was making sure that the workers completed without incident - or that any errors were caught so that the job could be 'bury'ed, thus ensuring that the job was not put back into the queue to be run again (and have the worker crash again).

Having the workers also restart to clear memory can also be useful as PHP is less well suited to long-running processes.

Stubblefield

What was used to pull jobs off the queue and execute them? Was this also done in PHP? If so, were the workers executed using something like System_Daemon or was there a cron job that ran every once and awhile? Other approaches? – Ainu

I start a shell script with Supervisord, and it keeps itself rerunning, waiting for a new item, then going again. There's a post with some more details at my blog - phpscaling.com/2009/06/23/… – Stubblefield