How to count the number of keys matching a pattern?

Question

How can I find the count of all the keys that has a matching pattern.

For example, there are two keys abc:random-text-1 and abc:random-text-2 . The common pattern here isabc: . So, here the count is 2.

How can I do this in redis?

Ecthyma

Counting keys in the Redis in production is a bad idea, forever. – Mineralize

Answer 1

DISCLAIMER I hope this old answer haven't damaged any production systems, with millions of keys. If you still want to still count the matching keys of redis in production for some reason, better use scan with a match pattern.

If you simply search with KEYS, with your redis client, you will get a number list of all you matching keys, right?

e.g.

KEYS abc:*

will give you

1) abc:random-text-1
2) abc:random-text-2

or you can run the following:

./redis-cli KEYS "abc:*" | wc -l

and you will get 2 as an output.

Dianemarie

That's exactly what I needed ! Thanks – Dangerous

And what if one has millions of keys? – Jaxartes

That's an old answer. I should actually add a disclaimer here. So do not use this in a production setup with millions of keys. Better use the SCAN cursor with MATCH – Dianemarie

You should never run this command on production, this is the blocking command and if you run this command all request will starve and may get timeout if this command runs longer. – Helvetian

Thanks! I noticed as well and added a disclaimer on top, a year ago. When I answered 6+ years ago I didn't consider that someone might run this on production. – Dianemarie

@Dianemarie why only put a text disclaimer, I guess you could add the command related (./redis-cli --scan --pattern "abc:*" | wc -l), couldn't you? Or point to an answer giving this. – Sandry

Will this be advisable to do when the keys expire? for example, im trying to get the number of active users by storing to redis. that key has expiration. will it still affect the production? Thanks – Corolla

Hey, what if we need select database? – Appearance

Answer 2

From here:

eval "return #redis.pcall('keys', 'abc:*')" 0

It's not O(1), but at least the count is done on the server side.

Trude

Note that this still uses the blocking keys command, which can cause performance hiccups on large instances. – Hubblebubble

2 seconds for 600K keys, fast enough – Notepaper

That's O(n) where n is number of keys. Really bad for millions of keys! – Mineralize

Never(!!) call this in production, no matter how small your data set is! – Ario

@AnatoliiStepaniuk Never say "never". – Trude

@Trude unless you want to risk putting your system down. There is a safe option with SCAN command. Do you have any reason not to use it? – Ario

@AnatoliiStepaniuk I don't know. Everyone has his reasons. But saying "never (!!)" is an absolute statement. The correct approach is to explain your reasoning and allow people decide for themselves when they need to choose. (I think this answer was written before better options were available). – Trude

Answer 3

From the command line, redis-cli --scan --pattern 'abc:*' | wc -l

Anacoluthon

Your answer is perfect but need to explain in details eg. Your redis keys 127.0.0.1:6379> keys abc* 1) "abc123" 2) "abc456" 3) "abc234" Now try to access this: From the command line, redis-cli --scan --pattern 'abc*' | wc -l Response below: localhost@username~$ redis-cli --scan --pattern 'abc*' | wc -l 3 – Rickert

This should be the accepted answer. Requires redis with version >= 2.8.0 – Pulchritude

Do I understand correctly that all matching records are fetched to the client? – Trude

Answer 4

By considering the performance, I would not recommend you use KEYS

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.

I would suggest you considering scan, if your redis version > 2.8.0. But it rely on which data type you are going to use.

Here is an simple example from redis doc:

redis 127.0.0.1:6379> sadd myset 1 2 3 foo foobar feelsgood
(integer) 6
redis 127.0.0.1:6379> sscan myset 0 match f*
1) "0"
2) 1) "foo"
   2) "feelsgood"
   3) "foobar"

Killjoy

How would you use scan to count keys? – Chuckhole

Either your can read official doc: redis.io/commands/scan. Or google it: #33167312 – Killjoy

Well, Googling took me here :) It would be helpful if you could edit your answer to include a simple command using scan for counting the number of keys matching a pattern. – Chuckhole

okay, I take your suggestion. :-) It might be easier for the others too. – Killjoy

@Killjoy Looking at scan api does not look like you can find the total number of count in one attempt. – Gouda

If you want to count keys using SCAN, you'll need to store the keys on each iteration, and use your preferred method of filtering out duplicates (SCAN does not guarantee values will only appear once), probably in a data structure that allows fast lookup of existing values. For giant sets of keys, this will be impractical since you'll need to store all keys in memory until the SCAN is complete, so you'll need to consider either keeping a secondary key as a counter which you update on SET/DEL, or storing your data in a type that allows easy counting (e.g. in a Redis hash). – Hardwood

Answer 5

If it's a one-time thing, you can use KEYS as described by x_maras, but you shouldn't use that in your code since KEYS will scan every key in the entire database each time it's called.

If you want to do it frequently, there is no "good" way exactly as you've written because it will always be fairly inefficient to scan every key (even using SCAN, since it would be doing the same thing as KEYS just in a safer manner).

However, if the patterns you need are known ahead of time, you can keep a set of every key that matches the pattern.

SET abc:random-text-1 "blah"
SADD patterns:abc abc:randomtext-1

SET abc:random-text-2 "more blah"
SADD patterns:abc abc:randomtext-2

SCARD patterns:abc
// (integer) 2

SORT patterns:abc BY nosort GET *
// 1) "blah"
// 2) "more blah"

Braca