beanstalkd - what happens to reserved, but not completed jobs?

Question

I've created a PHP script that reads from beanstalkd and processes the jobs. No problems there.

The last thing I've got to do is just write an init script for it, so it can run as a service.

However, this has now raised another question for me. When trying to stop the service, the one obvious way of doing it would be to try and kill the process. However, if I do that, what will happen to the job, if the PHP script was halfway through processing it? So the job was reserved, but the script never succeeded or failed (to delete or bury respectively), what happens?

My guess is that the TTR will expire, and then it gets put back to the ready queue?

And bonus 2nd question, any hints on how to better manage stopping the PHP service?

Orchardman

Answer 1

Any job that runs out of time, and is not buried or touched goes back into the ready queue to be reserved.

I've posted elsewhere about using Supervisord and shell scripts to run workers. It has the advantage that most of the time, you probably don't mind waiting for a little while as jobs finish cleanly. You can have supervisord kill the bash scripts that run a worker script, and when the script itself has finished, simply exits, as it can't be restarted.

Another way is to put a highest-priority (0) message into a tube that the workers listen of, that will have the workers first delete the message, and then exit. I setup the shell scripts to check for a specific return value (from exit($val);) and then they too would exit any loop in the shell scripts.

I've used these techniques for Beanstalkd and also AWS:SQS queue runners for some time, dealing with millions of jobs per day running through the system.

Ferd

Thanks for the answer Alister! I have already started with allowing an "exit" command to be sent, in order to end a worker. Probably worth looking into supervisord as well, since at the moment my process handling does feel a bit flimsy! – Orchardman

Answer 2

When a worker process (beanstalk client) opens up a connection with beanstalkd and reserves a job, the job will be in "reserved" state until the client issues delete/release command (or) job times out.

In case, if the worker process terminates abruptly, its connection with beanstalkd will get closed and the server will immediately release all the jobs that has been reserved using that particular connection.

Ref: http://groups.google.com/group/beanstalk-talk/browse_thread/thread/232d0cac5bebe30f?hide_quotes=no#msg_efa0109e7af4672e

Slavey

Answer 3

If you job is too valuable to lose, you can also use pcntl to wait until the job finishes and then restart/shutdown your worker. I've managed to handle all suitable pcntl signals to release the job back to tube.

Iand