Redis: Show database size/size for keys

Question

My redis instance seems to being growing very large and I'd like to find out which of the multiple databases I have in there consumes how much memory. Redis' INFO command just shows me the total size and the number of keys per database which doesn't give me much insight... So any tools/ideas that give me more information when monitoring the redis server would be appreciated.

The Redis documentation doesn't show me any commands that can return the consumed memory of certain keys, so I guess if any buggy code would write a lot of "trash" to redis this could be really hard to find...

Bandit

Answer 1

So my solution to my own problem: After playing around with redis-cli a bit longer I found out that DEBUG OBJECT <key> reveals something like the serializedlength of key, which was in fact something I was looking for...

For a whole database you need to aggregate all values for KEYS * which shouldn't be too difficult with a scripting language of your choice...

The bad thing is that redis.io doesn't really have a lot of information about DEBUG OBJECT.

Bandit

Is serialized length the size of the entire object, in bytes? – Eutherian

@BernhardVallant, thanks for answer. I went ahead and wrote a quick script that prints all the keys and their sizes in a human readable format. Thought I would share. :) gist.github.com/epicserve/5699837 – Praise

The value of serializedlength is not about memory size! It is the size an objects would take when save to a RDB file on disk. Check the source code: github.com/antirez/redis/blob/… and github.com/antirez/redis/blob/… – Christianity

there doesn't seem to be a way to get the size of the key or value or key-value in memory (Not the dumped size to disk) – Fernand

This does not include the size of the key itself, only the value. – Carnify

FYI: don't bother trying anything with DEBUG on AWS ElastiCache Redis, docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/…. Even found redis-cli --bigkeys would stall – Stereograph

You can take script created by @BrentO'Connor and replace "debug object" by "memory usage". Works for ElasticCache where "debug" is disabled – Attach

Answer 2

MEMORY USAGE key command gives you the number of bytes that a key and its value require to be stored in RAM.

The reported usage is the total of memory allocations for data and administrative overheads that a key its value require (source redis documentation)

Stagecraft

this can only be used version>4.0 – Smuts

Answer 3

The solution from the comments deserves its own answer:

redis-cli --bigkeys

Cleanlimbed

big keys is about the size of the key, not the size of the value stored .. so you could have a key of a with a 4GB value but this would not show up in bigkeys. (this is going by gist.github.com/michael-grunder/9257326 and docs.redislabs.com/latest/ri/memory-optimizations/… ) – Kinesthesia

--bigkeys does an object scan and tells you by content, not just the key length – Flitting

Answer 4

Take a look at this project it outputs some interesting stats about keyspaces based on regexs and prefixes. It uses the DEBUG OBJECT command and scans the db, identifying groups of keys and estimating the percentage of space they're taking up.

https://github.com/snmaynard/redis-audit

Output looks like this:

Summary  

---------------------------------------------------+--------------+-------------------+---------------------------------------------------  
Key                                                | Memory Usage | Expiry Proportion | Last Access Time                                    
---------------------------------------------------+--------------+-------------------+---------------------------------------------------  
notification_3109439                               | 88.14%       | 0.0%              | 2 minutes                               
user_profile_3897016                               | 11.86%       | 99.98%            | 20 seconds  
---------------------------------------------------+--------------+-------------------+---------------------------------------------------  

Or this this one: https://github.com/sripathikrishnan/redis-rdb-tools which does a full analysis on the entire keyspace by analyzing a dump.rdb file offline. This one works well also. It can give you the avg/min/max size for the entries in your db, and will even do it based on a prefix.

Guitarfish

Answer 5

You might find it very useful to sample Redis keys and group them by type. Salvatore has written a tool called redis-sampler that issues about 10000 RANDOMKEY commands followed by a TYPE on retrieved keys. In a matter of seconds, or minutes, you should get a fairly accurate view of the distribution of key types.

I've written an extension (unfortunately not anywhere open-source because it's work related), that adds a bit of introspection of key names via regexs that give you an idea of what kinds of application keys (according to whatever naming structure you're using), are stored in Redis. Combined with the more general output of redis-sampler, this should give you an extremely good idea of what's going on.

Rundgren

Tks, actually helped me more than the redis-cli --bigkeys – Aldous

Answer 6

You can also check the INFO command in redis to view the memory usage

$ redis-cli
127.0.0.1:6379> INFO memory

Namtar

this will provide overall memory info but not at the database level which OP is asking for. – Chickaree

Answer 7

Perhaps you can do some introspection on the db file. The protocol is relatively simple (yet not well documented), so you could write a parser for it to determine which individual keys are taking up a lot of space.

---

New suggestions:

Have you tried using MONITOR to see what is being written, live? Perhaps you can find the issue with the data in motion.

Imbecility

Seems to be interesting, but nevertheless I'my trying to find an easy way to monitor redis' memory consumption on the server... Examining the dump seems to be more practicable for debugging to me, not too mention that the dump is a few gigs now! – Bandit

You should ask the redis mailing list. I'm really interesting in hearing the "best" answer for this. – Imbecility

Well have already tried INFO and MONITOR, but may main problem is, that when not watching redis grew really big... – Bandit

Ok i posted it to their mailing list, but found an answer on my own also... See below! – Bandit

re. introspection on db file - I wrote a script to parse dump.rdb files and output a csv file reporting the approximate memory used by each key. See github.com/sripathikrishnan/redis-rdb-tools – Alber

Answer 8

Quick and dirty if you know the prefix you can sum the set of keys:

echo "keys userfeed*" | redis-cli -h 10.168.229.48 | xargs -I{} echo "debug object {}" | redis-cli -h 10.168.229.48 | awk '{print $5}' | cut -
d: -f2 | awk '{s+=$1} END {print s}'

This give the size in bytes. Remember this is "serialized length" not size in memory.

To get size in memory in bytes:

echo "keys op*" | redis-cli -h 10.168.229.48 | xargs -I{} echo "memory usage {} samples 0" | redis-cli -h 10.168.229.48 | awk '{s+=$1} END {pr
int s}'

Brow

Answer 9

You're Right the Redis documentation doesn't show any commands that can return the consumed memory of certain keys. Redis does not provide an out-of-the-box command to retrieve memory consumption per database or per key. But don't worry there are many ways

1. A very simple solution :

Use the redis-cli cmd tool with --bigkeys see below cmd

redis-cli --bigkeys

2. You can use the MEMORY USAGE cmd that returns the number of bytes consumed by a key see below cmd.

   redis-cli MEMORY USAGE <your-redis-key>

3. We have best solution to use external tools to monitor and analyze Redis instance.

Use RedisInsight A GUI-based Redis management tool that allows to inspect and manage Redis instances, including memory analysis.

See the image I use RedisInsight to monitor db. RedisInsight GUI is user-friendly

These are the methods, that I used to identify the memory consumption patterns in Redis instance and track down any problematic keys when I'm working. and I hope these methods assist you too.

Ricciardi

Answer 10

I usually prefer the key sampling method to troubleshoot such scenarios.

redis-cli -p 6379 -n db_number --bigkeys

Eg:-

redis-cli -p 6370 -n 0 --bigkeys

Wyoming

Why is this the "key sampling" method? I find it very limited as it only shows the tip of the iceberg. – Schoolroom

Answer 11

lua one-liner

eval "local sum = 0; local matches = redis.call('KEYS', '<pattern>'); for _,key in ipairs(matches) do local val = redis.call('memory', 'usage', key); sum = sum + tonumber(val) end return sum" 0

Rosewater

Answer 12

You can use .net application https://github.com/abhiyx/RedisSizeCalculator to calculate the size of redis key,

Please feel free to give your feedback for the same

Gladiolus

Answer 13

How about

redis-cli get KEYNAME | wc -c  

Solfatara

Answer 14

One-liner. All redis keys and corresponding used memory.

redis-cli keys "*" | while read line; do echo -n "$line: "; redis-cli memory usage $line; done

Harmonics