Redis block-push until list has vacant spot

Question

I'm seeking something like a BLPUSH command that will block until the length of a list drops below a specified max_size. The purpose is to prevent the list from growing indefinitely if the producer runs faster than the consumer.

The functionality will be very similar to python's Queue.put().

Note that this accepted answer does not work, because (1) the code returns immediately, which is not blocking at all. (2) if I write a spin lock in Lua, it will freeze up Redis.

Is there a way to implement BLPUSH in Lua, or does it have to be in C?

Befoul

Answer 1

You can implement this using 2 lists and BRPOP/BLPOP. The accompanying tokens list tracks available space in the queue list.

Example pseudo-code assuming a maxsize of 3 elements (just to make the example concrete).

DEL queue, tokens   # start with 2 keys, delete to start with a clean slate
LPUSH tokens 0 0 0  # push maxsize tokens to the tokens list (3 in this example)

On put(item):
  BRPOP tokens      # remove a token when pushing, this blocks if no tokens exist
  LPUSH queue item  # this is safe because tokens keeps size count

On get():
  LPUSH tokens 0    # add a token when taking from the queue
  BRPOP queue       # take from the queue, block when empty

Intermediary

Answer 2

TLDR it is has to be in C, or in any other language with C bindings.

Lua is a means for composing logic and existing Redis commands. As Redis is (mostly) single-threaded, the Lua runtime engine gets full attention for running the script and blocks the server.

As of Redis v4, modules allow anyone (that has the required dev skillz) to extend Redis with (almost) anything, including blocking operations.

Burnell