how to get keys which does not match a particular pattern in redis?

Question

In Redis, keys user* will print all keys starting with user. For example:

keys user*
1) "user2"
2) "user1"

Now, I want all keys that don't start with user to be printed. How could I do that?

Jordanson

What is your use case for this? – Pamulapan

i want to delete keys which does not match the given pattern. I got the lua script for deleting keys which matches the given pattern. I don't know how to get the keys which are not matching a particular pattern. – Jordanson

say i l be storing keys wit date appended like set 27_Apr_2015_result_1 "sachin" set 28_Apr_2015_result_2 "Dravid" set 29_Apr_2015_result_1 "David" now i don't want my redis to store things stored everything before 29. Only way i could do is getting the keys that doesnot match 29_Apr_2015*. This is my use case @Tim Cooper – Jordanson

@KarthikeyanGopall Can you show us the Lua-script for deleting keys matching a pattern? It might be a good base to change it do to the opposite. – Madore

script="redis.call('del',unpack(redis.call('keys', '%s')))"; %s will be replaced by the pattern while this function is called @Madore – Jordanson

Answer 1

IMPORTANT: always use SCAN instead of (the evil) KEYS

Redis' pattern matching is somewhat functionally limited (see the implementation of stringmatchlen in util.c) and does not provide that which you seek ATM. That said, consider the following possible routes:

1. Extend stringmatchlen to match your requirements, possibly submitting it as a PR.
2. Consider what you're trying to do - fetching a subset of keys is always going to be inefficient unless you index them, consider tracking the names of all non-user keys (i.e.g. in a Redis Set) instead.
3. If you are really insistent on scanning the entire keyspace and match against negative patterns, one way to accomplish that is with a little bit of Lua magic.

Consider the following dataset and script:

127.0.0.1:6379> dbsize
(integer) 0
127.0.0.1:6379> set user:1 1
OK
127.0.0.1:6379> set use:the:force luke
OK
127.0.0.1:6379> set non:user a
OK

Lua (save this as scanregex.lua):

local re = ARGV[1]
local nt = ARGV[2]

local cur = 0
local rep = {}
local tmp

if not re then
  re = ".*"
end

repeat
  tmp = redis.call("SCAN", cur, "MATCH", "*")
  cur = tonumber(tmp[1])
  if tmp[2] then
    for k, v in pairs(tmp[2]) do
      local fi = v:find(re) 
      if (fi and not nt) or (not fi and nt) then
        rep[#rep+1] = v
      end
    end
  end
until cur == 0
return rep

Output - first time regular matching, 2nd time the complement:

foo@bar:~$ redis-cli --eval scanregex.lua , "^user"
1) "user:1"
foo@bar:~$ redis-cli --eval scanregex.lua , "^user" 1
1) "use:the:force"
2) "non:user"

Panoply

but i want an atomic operation that is the reason i opted keys rather than scan. I thought there might be some way that i could get the keys that doen't match the pattern with keys command. Now got to know tat it's not possible. Thanks for ur explanation @Itamar Haber – Jordanson

A Lua script is atomic – Panoply

If you wanted to search for fixed strings, ARGV[1]:gsub('(['..("%^$().[]*+-?"):gsub("(.)", "%%%1")..'])', '%%%1') can be used to escape special characters. – Celka

Answer 2

@Karthikeyan Gopall you nailed it in your comment above and this saved me a bunch of time. Thanks!

Here's how you can use it in various combinations to get what you want:

redis.domain.com:6379[1]> set "hello" "foo"
OK
redis.domain.com:6379[1]> set "hillo" "bar"
OK
redis.domain.com:6379[1]> set "user" "baz"
OK
redis.domain.com:6379[1]> set "zillo" "bash"
OK
redis.domain.com:6379[1]> scan 0
1) "0"
2) 1) "zillo"
   2) "hello"
   3) "user"
   4) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u]*"
1) "0"
2) 1) "zillo"
   2) "hello"
   3) "hillo"
redis.domain.com:6379[1]> scan 0 match "[^u^z]*"
1) "0"
2) 1) "hello"
   2) "hillo"
redis.domain.com:6379[1]> scan 0 match "h[^i]*"
1) "0"
2) 1) "hello"

Flare

Answer 3

According to redis keys documentation the command supports glob style patterns, not regular expressions.

and if you look at the documentation, you'll see that the "!" character is not special as opposites to regular expressions.

Here is a simple test I ran in my own db:

redis 127.0.0.1:6379> set a 0
OK
redis 127.0.0.1:6379> set b 1
OK
redis 127.0.0.1:6379> keys *
1) "a"
2) "b"
redis 127.0.0.1:6379> keys !a   
(empty list or set)                       // I expected "b" here
redis 127.0.0.1:6379> keys !b
(empty list or set)                       // I expected "a" here
redis 127.0.0.1:6379> keys [!b]
1) "b"
redis 127.0.0.1:6379> keys [b]
1) "b"
redis 127.0.0.1:6379> keys [ab]
1) "a"
2) "b"
redis 127.0.0.1:6379> keys ![b]
(empty list or set)

So I just don't think what you are trying to achieve is possible via the keys command.

Besides, the keys command is not very suitable for production environment as it locks your whole redis database.

I would recommend getting all the keys with the scan command, store them locally, and then remove them using LUA

Skat

i guess tat's d only option i left with :( thought of something else may be there to solve this issue. However u can try doing this with ^ instead of !. That will work for only one character and not for the whole string. That's wat actually the main problem is. Example: u hav keys like hello, hallo, hi. if i perform keys [^he]* it should show hallo and hi but it shows nothing as it takes the negation for 1st character alone. – Jordanson

Answer 4

[^host|^redis]*:*

Exludes keys starting with host and redis and returns all patterns with :.

Proudfoot

This just checks if the first character is not one of 'h', 'o', 's', 't', '|', '^', ... only very few glob-style patterns are supported and they are evaluated character-wise, so [^hr][^oe] also doesn't do it. – Committeewoman

Answer 5

Here's a trick to achieve this with native redis commands (no need for Lua scripts or anything).

If you are able to control the timing of when you insert the new keys (the ones you want to keep, deleting all other stuff like in your question), you can:

1. Before setting the new keys, set the expiration to all existing keys (by pattern or everything) to expire right now (see how)
2. Load the new keys

Redis will automatically delete all the older keys and you will be left just with the new ones you want.

Pointsman

Answer 6

You also can print all keys and pass it to grep. For example:

redis-cli -a <password> keys "*" | grep -v "user"

Perreault

Answer 7

leppaott's answer + the Redis docs helped me.

I wanted to run a scan to pull back my top level hashes ( called zone:foobar ) while excluding sub-hashes ( called zone:foobar:plan ) and other things I had set up.

# all
SCAN 0 MATCH *zone*
1) "0"
2) 1) "zone:foobar:plan"
   2) "zone:foobar"
   3) "pipelined_counter:zone"

# exclude sub hashes
SCAN 0 MATCH zone:*[^plan]
1) "zone:foobar"

The thing I like about the above, versus some internet answers, it uses the Glob syntax from Redis with all the logic stay on the Redis side versus extracting all keys and filtering in my app code.

Cris