Is it ok to use PHP-FPM to manage queue consumers?

Question

There's a beanstalkd queue, which gets filled with a lot of tasks say every 10 minutes and it is top priority that each task is processed ASAP. Task can take more that a few milliseconds to complete for there are calls to third-party services, which tend to timeout every now and then.

So since PHP doesn't have multithreading, one option would be to create a lot of idle workers, which would try to reserve a task, but it is likely to take too much RAM, which may not be available on those boxes.

Is it a good idea to use PHP-FPM to adjust the number of workers and save some RAM? Is it production-ready? Are there better solutions?

Thanks

Guyette

Answer 1

I'm running a queue system that is dealing with millions of messages per day. Mostly via Amazon SQS, but I'm also running a new Beanstalkd system with over 600,000 msgs in there right now.

As is described in a blogpost on the subject, I have shell scripts running in a loop processing messages (a loop within the PHP script to run multiple jobs before returning is also somewhat useful, at least for smaller jobs).

Those shell scripts are started with Supervisord. There's another blog post on the use of that as well. I'm currently running over 800 worker scripts (for a few different types of jobs) across nine machines, all pulling from various queues and putting data back into other queues, writing to the DB or files. Increasing the number of workers per machine is a matter of increasing the "numprocs" (or having it large enough already), and then starting more as required. You could also have say 5 auto-start, and then another block of 50 that are ready to start as required.

I find each worker only takes around 20mb of non-shared memory (the rest is common between processes). This does depend on the tasks that the workers do of course. Resizing images can take a lot of effort. It's partly for this reason I have setup to be able to frequently restart the PHP script.

Bimetallic

Answer 2

Whenever I had to run stuff concurrently (or asynchronously), I dispatched the jobs to gearman workers. I usually had at least one process per CPU core per physical machine running.

PHP-FPM is a cgi daemon. So you'd basically have your beanstalkd-processor run a bunch of HTTP requests to your own system. Those would probably have to go through your http stack. Not sure if that is such a great idea.

You could also check out pcntl_fork to fork your current process into multiple cuncurrently running processes.

Pole

Thanks for the suggestion, I'll have a look at gearman. There's a PHP client for FPM, which allows to bypass HTTP stack, so that's somewhat better, but not perfect still. And forking would basically consume the same memory as just running different processes would, right? – Guyette

Yes forks consume their own memory. But they're only created when needed and removed afterwards. With PHP-FPM you always keep a bunch of processes cached (which makes total sense…). hm, I didn't know about any FPM client, where can it be found? – Pole