Simple scalable work/message queue with delay

Question

I need to set up a job/message queue with the option to set a delay for the task so that it's not picked up immediately by a free worker, but after a certain time (can vary from task to task). I looked into a couple of linux queue solutions (rabbitmq, gearman, memcacheq), but none of them seem to offer this feature out of the box.

Any ideas on how I could achieve this?

Thanks!

Jackshaft

Answer 1

I've used BeanstalkD to great effect, using the delay option on inserting a new job to wait several seconds till the item becomes available to be reserved.

If you are doing longer-term delays (more than say 30 seconds), or the jobs are somewhat important to perform (abeit later), then it also has a binary logging system so that any daemon crash would still have a record of the job. That said, I've put hundreds of thousands of live jobs through Beanstalkd instances and the workers that I wrote were always more problematical than the server.

Hallucinate

Thank you for your feedback, I've been looking into beanstalkd the past couple of days and it looks great. And you're right, worker logic and management is tricky :) – Jackshaft

Answer 2

You could use an AMQP broker (such as RabbitMQ) and I have an "agent" (e.g. a python process built using pyton-amqplib) that sits on an exchange an intercepts specific messages (specific routing_key); once a timer has elapsed, send back the message on the exchange with a different routing_key.

I realize this means "translating/mapping" routing keys but it works. Working with RabbitMQ and python-amqplib is very straightforward.

Escaut

I thought about this, but if the waiting agent gets killed while it's waiting for the timer to elapse, the message never gets added to the queue. Maybe I could fix this by having a second, permanent queue, and an agent with multiple internal timers. Still, it seems like an ugly workaround. – Jackshaft

Well, dealing with this failure-mode is part of the game. I would be surprised if the functionality you are asking gets mainstream in AMQP Brokers out-there since it goes against the main goal: minimize latency. – Escaut

Thank you for your replies. I think I will give beanstalkd a try, it seems ok, and supports the "delay" I was talking about. – Jackshaft