Check-and-increment a counter in Redis

Question

I have an operation I need to get done N times and no more. The operation is done by many possibly parallel processes that recieve requests. A process has to check if the counter has exceeded N and, if not, then increment the counter and execute the operation. I suppose I could use a Redis counter for that.

Howerver if I just GET and then INCR a value I might run into a race condition that will result in the operation being done more than N times. How do I perform some kind of test-and-incr operation against Redis?

I know I could use WATCH but that's an optimistic lock. I expect there going to be very many collisions each second. This will result in a lot of failures. Maybe I could just wrap simple GET and INCR with some kind of external mutex. But I am not sure if it's good enough for performance.

Primeval

Just do it in Lua which will happen atomically. – Ungraceful

Or put N items in a list and do BLPOPs off the list. – Ungraceful

Answer 1

• You can use Redis INCR safely here.

• INCR returns the value post the increment.

• You check the value returned by INCR first ( see there is no need to do a GET ) & proceed to do the operation based on that value.

• Only thing is you would have to set your INCR return value threshold as N+1 for limiting to N operations i.e. one extra redis operation than N. For example, if we want to limit the operation to happen 3 times only, if INCR returns 4 after an increment, then you stop doing further operation as it has already happened 3 times.

Rhesus

If I INCRed when it appears I shouldn't have then I can just follow it with DECR, right? – Primeval

Really loved the idea. What If we set the initial count to zero, the N + 1 value will be handled automatically ? – Theolatheologian

Answer 2

Try this

Increment

SET my_counter 10
INCR my_counter   // Result: 11
INCR my_counter   // Result: 12

Increment and return new value

// INCR command increments the value of the key and returns the new value
const incrementedValue = await redis.incr(key);
console.log("Incremented value:", incrementedValue);

Sitter

Answer 3

Here is an example of how to do this with single LUA script, as Redis is signle-threaded you don't need to add any locks or anything else here.

-- This script increments a counter and returns a boolean indicating whether the counter is below a limit. --
-- If the counter is below the limit, the counter is incremented and the new value is returned. --
-- If the counter is above the limit, the counter is not incremented and the current value is returned. --
-- The counter is set to expire after 60 seconds. --
-- ARGV[1] - The key of the counter in Redis --
-- ARGV[2] - The command INCR or DECR --
-- ARGV[3] - Limit to check against --

local counter
local counter_key = ARGV[1]
local command = ARGV[2]
local limit = tonumber(ARGV[3])
local expiration_in_seconds = 60
local success = false

counter = redis.call("GET", counter_key)
if counter == false then
  counter = 0
end

if tonumber(counter) < limit and command == "INCR" then
  success = true
  counter = redis.call("INCR", counter_key)
elseif tonumber(counter) > limit and command == "DECR" then
  success = true
  counter = redis.call("DECR", counter_key)
end

redis.call("EXPIRE", counter_key, expiration_in_seconds)

return { success, tonumber(counter) }

-- Example of call --
-- EVAL ${LUA_SCRIPT_ABOVE_AS_STRING} 0 test_key INCR 10 --

Example or client in Ruby (you can reimplement in your language of choice):

module RedisHelpers
  module CounterWithLimit
    LUA_SCRIPT = File.read("#{__dir__}/counter_with_limit.lua").freeze
    # NOTE: Redis returns 1 for true and nil for false when LUA script is executed
    SUCCESS_VALUE = 1

    module_function

    def try_incr(key:, limit:, redis: Redis.new)
      # NOTE: https://redis.io/commands/eval/
      success, counter = redis.call('EVAL', LUA_SCRIPT, 0, key, 'INCR', limit)

      { success: success == SUCCESS_VALUE, counter: }
    end

    def try_decr(key:, limit: 0, redis: Redis.new)
      # NOTE: https://redis.io/commands/eval/
      success, counter = redis.call('EVAL', LUA_SCRIPT, 0, key, 'DECR', limit)

      { success: success == SUCCESS_VALUE, counter: }
    end
  end
end

Incudes