Redis: how to delete multiple keys matching pattern?

Question

I need to remove 10 000 keys.

1. What is better way: to exec this kind of script

   EVAL "return redis.call('del', unpack(redis.call('keys', ARGV[1])))" 0 "ROOT"

2. May be better is to set expiration time and Redis will remove them? But how to do it in console using Lua script?

The script (see above) works because del comman dhas a format:

del key1 key2 ...

But Expire works only for 1 key.

Is it possible to do it in lua script?

For example: my application creates some search results cache and set for every page ttl = 3600. But user wants to clear cache immediately, i.e. delete all matching keys or set smaller expiration for them.

Decrescent

Before recommending on an approach for doing this, do you need all keys delete in one atomic batch or are your requirements more lenient? – Parish

I'd like to execute this command (batch?) in c# application which has GUI. – Decrescent

So you are aware that an atomic operation will block the Redis server while it is running and that's inline with the requirements (i.e. don't run it in an environment where Redis is needs to remain available to server requests), right? – Parish

Yes, you are right. But I do not want to run loop in my client application, because it will be not optimal and long operation. I need to set expiration for multiple keys matching some pattern. I see that using 'keys' in my example is not a good solution too. What else? – Decrescent

Answer 1

Whether you DEL or EXPIRE, once Lua script runs it will block other clients and if it runs for too long it the lua-time-limit timeout. Despite your reluctance for looping, I strongly recommend you do.

Expiry vs deletion may lessen some of the immediate load (yet to be empirically proven), so feel free to go with one or the other. In either case, use a client-side loop on a SCAN operation to invoke the command for each key. If you have a server/worker process somewhere in your architecture, you can consider delegating this task to it so the client will not be kept busy.

EDIT per comment: Variadic commands such as DEL are generally more performant than non-variadic commands, however here you're comparing two different operations so there are no assurances. The DEL approach is potentially more blocking because Redis will actually go ahead and delete the keys immediately - if you have a lot of keys to delete and/or their values are big, this could take more time. The EXPIRE approach tries to avoid this by leveraging on Redis' lazy expiry mechanism (it uses idle time to do so when possible), so the deletion-due-to-expiry load is theoretically better distributed. The best way to determine which works better for you is by testing both and comparing - I'd love to learn of your results!

Parish

ok, thnx. Yet another question. My task is to remove keys. Ok, 1) I use SCAN and call EXPIRE for each record. It is clear. 2) But instead of that I can call DEL for subset of keys from loop. What is faster, more optimal and less locking? – Decrescent

Answer 2

You can either use (from redis cli) to delete all keys:

flushall

or run this from your command line (bash)

redis-cli --scan --pattern aihello_user* | xargs redis-cli del 

Antimissile

the second command should be the accepted answer. It will delete only the keys matching the provided pattern, without affecting the rest of the data – Turnip

I would use unlink instead of del as it is non-blocking. Explanation : redis.io/commands/unlink – Insentient

note the curly braces: redis-cli --scan --pattern aihello_user* | xargs redis-cli del {} – Giovanna

Using MacOS with zsh, I had to add quotation marks to the pattern, like this redis-cli --scan --pattern "workflow-status-*" | xargs redis-cli del – Cultivator

Answer 3

if you are trying to delete a key matching a prefix then you can try below command

redis-cli keys <PREFIX>'*' | xargs redis-cli del

here keys <PREFIX> '*' will give all the keys with matching prefix and then del command will delete all of them.

Ditto

Answer 4

redis-cli -h 127.0.0.1 -p 6379 --scan --pattern <pattern>* | xargs redis-cli unlink 

Using scan and unlink are not blocking functions

Subassembly

Answer 5

I was looking all over for this ended up writing a bash script that takes in a host, port, and a pattern. Seems to work well. Here is the Gist.

if [ $# -ne 3 ] 
then
  echo "Delete keys from Redis matching a pattern using SCAN & DEL"
  echo "Usage: $0 <host> <port> <pattern>"
  exit 1
fi

cursor=-1
keys=""

while [ $cursor -ne 0 ]; do
  if [ $cursor -eq -1 ]
  then
    cursor=0
  fi

  reply=`redis-cli -h $1 -p $2 SCAN $cursor MATCH $3`
  cursor=`expr "$reply" : '\([0-9]*[0-9 ]\)'`
  if [[ $reply = *[[:space:]]* ]]
  then
    keys=${reply#[0-9]*[[:space:]]}
    for key in $keys; do
      echo "Delete the following key: $key"
      redis-cli -h $1 -p $2 DEL $key 
    done
  fi
done

Gun

Answer 6

Here's how you can expire in bulk via the CLI:

EVAL 'for i, name in ipairs(redis.call("KEYS", <pattern>)) do redis.call("EXPIRE", name, 1); end' 0

Wiser