Delayed execution / scheduling with Redis?

Question

Any tricks to do delayed task execution (i.e. scheduling) based on Redis?

Perhaps some clever way to delay BLPOP for a given number of seconds?..

Orthoclase

Answer 1

You can work with a ring of multiple LISTs that have a time component in their name. As time component you can take the current second (0-59).

You always add tasks to the list for the current second. To get the jobs you do a BLPOP (with low timeout) only on those lists where it is guaranteed, that the content is older than the given number of seconds.

If you work from multiple hosts you have to take care, that the clocks are in sync (NTP).

Scala

Answer 2

If you want to do scheduling with redis, i would suggest using sorted set (the z*) commands:

http://code.google.com/p/redis/wiki/SortedSets

what you can do is something like this:

ZADD jobs <unix timestamp of when you want the job to run> <job identifier>

e.g:

ZADD jobs 1291348355

Then, every so often (up to every second) you can pull scheduled jobs that should run (or should have run by now):

ZRANGEBYSCORE jobs -inf, <current unix timestamp>

Boom, you got your jobs to run. Of course, make sure to delete done jobs from the sorted set.

Bandmaster

Cool, but I'd like to get BLPOP semantics. That is, I need my query to Redis to be blocking until I've got something to run. One second granularity is not very robust. – Orthoclase

are you more scheduling or are you queueing? I guess it depends on the precision at what you are working. When I think of scheduling jobs I think in terms of "in 2 hours" or "every hour on the hour" etc. I don't see sub-second scheduling precision being an issue in that use case. Queueing/backgrounding sounds more like what you are looking for, that is: "i want to push jobs to the background and having workers ready to go as soon as they are queued." In this instance, BLPOP makes a ton of sense. I use methods like this when background processing streaming data (twitter content, for example). – Bandmaster

Oh, old thread but - Alexander - you could use kind of dispatcher thread which checks sorted sets every N seconds and just move jobs from sorted set to main job list while main job workers are blpop'ing. – Eubanks

BLPOP approach is easy to scale one. You could have multiple worker processes blocked at BLPOP waiting for new items. New items will be distributed among listeners. With ZRANGEBYSCORE approach it is tricky to have multiple workers. – Joellejoellen

Answer 3

While the @efalcao's answer is a very good one, your question might indicate that redis does not perfectly suit your application needs. if your application has the nature of a message box, please consider using rabbitMQ, which features delayed messages or akka if you feel bold

Cherianne