How to export all keys and values from memcached with python-memcache?

Question

I would like to export all keys and values from a memcached server, using python-memcache. There is no such function in that module. How to do it then?

Perhaps something more complicated involving the "socket" module would be needed.

Aspirant

Answer 1

This will get you all the keys on a memcached server, you can use any memcached client library to get the value for each key.

import telnetlib

def get_all_memcached_keys(host='127.0.0.1', port=11211):
    t = telnetlib.Telnet(host, port)
    t.write('stats items STAT items:0:number 0 END\n')
    items = t.read_until('END').split('\r\n')
    keys = set()
    for item in items:
        parts = item.split(':')
        if not len(parts) >= 3:
            continue
        slab = parts[1]
        t.write('stats cachedump {} 200000 ITEM views.decorators.cache.cache_header..cc7d9 [6 b; 1256056128 s] END\n'.format(slab))
        cachelines = t.read_until('END').split('\r\n')
        for line in cachelines:
            parts = line.split(' ')
            if not len(parts) >= 3:
                continue
            keys.add(parts[1])
    t.close()
    return keys

Illbehaved

A colleague is getting "zero length field name in format" at the line "t.write('stats cachedump...". Probably for earlier versions of Python you need "stats cachedump {0} 200000". – Tartuffery

It works for me in local server BUT in production environment it is not usable because it gets too slow (you can wait for many minutes) – Capita

Answer 2

use the memdump and memcat utilities from the libmemcached suite. They can't guarantee you'll get all the data but they're easy to use.

Note: On ubuntu/debian you can get these by intstalling the libmemcached-tools package and they are called memcdump and memccat.

dump all keys:

memcdump --servers=localhost

dump all values:

memccat --servers=localhost `memcdump --servers=localhost`

of course you still have to match up the keys and values - I'd suggest dumping the keys to a file and then using that as the input for memcat (this ensures consistency). Then of course you need to split the values - a full stop is the delimiter I believe - and then pair the keys and values sequentially. There's probably a script out there somewhere...

Scene

The memdump command is called "memcdump" in Ubuntu Precise, in the package "libmemcached-tools" – Schulz

yeah, memdump is completely different manpages.ubuntu.com/manpages/lucid/man1/memdump.1.html – Radionuclide

Answer 3

There is no way to do that. Memcache protocol does not define any command to iterate over keys. You have to know the key to retrieve value.

Imide

Still, via telnet you can run "stat items" and then "stats cachedump" Too bad the functions do not exist in python-memcache – Aspirant

@Falken: true, but that doesn't give you information about individual items, just aggregated stats. – Imide

@Falken: As for "cachedump" it's not guaranteed to work, it's meant for debugging only, return is limited to 1MB – Imide

Answer 4

The easiest way is to use python-memcached-stats package, https://github.com/abstatic/python-memcached-stats

The keys() method should get you going.

Multiplication

With Python3.8 that failed with an error at from memcached_stats import MemcachedStats. It's a pity as I was wanting a Pythonic way of doing this. – Psittacine

Above doesn't support python3+, feel free to fork and update or raise a PR! – Multiplication

OK, when I've finished porting your code to Python3, I'll come back here and up-vote this :). – Psittacine

Answer 5

As mentioned by others in many places, in the general case, there is no way to list all the keys that stored in a memcached instance. E.g., Memcached: List all keys, Couldn't retrieve all the memcache keys via telnet client

You can, however, list something like the first 1Meg of keys, which is usually enough to have an idea about what are stored in memcache server during development. Basically, you can have two option to extract items from memcache server:

(1) To retrieve a subset of keys and values, you can the method introduced above by use @lrd

However, when the data is very large (e.g., millions of records), this method can be very time-consuming. What's more, this method can only retrieve only a subset of the keys & values.

(2) In cases where you would like to iterate all the items of the memcached server, logging the keys when one adds/set/cas items to memcache server is a much cheaper solution. Then you can read through the log file to get all the keys and get the values from memcache server. As discussed in this mailing list: List all objects in memcached

Profundity

Answer 6

In Bash, this script may help:

while read -r key; do
    memccat --servers=localhost $key > "$key.dump";
done < <(memcdump --server localhost)

Instead of memccat, you can also connect to port directly and send get SOME-KEY command (example using Netcat: echo "get $key" | nc localhost 11211).

Related:

• get all keys set in memcached
• View content of memcached

Palsgrave

Answer 7

You're looking for the 'flush_all' memcache command: http://code.google.com/p/memcached/wiki/NewCommands#flush_all

With python-memcached, it looks something like this:

>>> import memcache
>>> c = memcache.Client(('127.0.0.1:11211',))
>>> c.flush_all()

Entertain

I think he means "dump" as in "export a list of" instead of "dump" as in "get rid of". – Crupper

Absolutely, my english may be a bit poor, I meant it as "export a list of" :D – Aspirant

this shouldn't be downvoted - it's an answer to the question as asked – Scene

The question has been corrected and so says "I would like to export all keys and values"; this has ended up being a misleading answer in its current form – Bedtime