How to use redis' `DUMP` and `RESTORE` (offline)?

Question

I tried redis's DUMP command, redirect to file (or pipe), but RESTORE report this error:

$ redis-cli dump test > /tmp/test.dump
$ cat /tmp/test.dump | redis-cli -x restore test1 0
(error) ERR DUMP payload version or checksum are wrong
$ redis-cli dump test | redis-cli -x restore test1 0
(error) ERR DUMP payload version or checksum are wrong

I am aware that MIGRATE can do this online, but MIGRATE also delete that key from original server, and I don't want my redis expose to public internet.

There are some third-party options, redis-rdb-tools for example, but after all, how exactly do DUMP and RESTORE work?

Mcwhorter

Answer 1

The dump/restore commands are not really designed to be used from the command line, because the serialization format is binary (it is the same one used for RDB dumps). It makes it inconvenient because the shell tends to interpret those characters (even when the "printable" format is used).

Here is the "printable" format:

$ redis-cli lpush test 1 2 3 4 5
(integer) 5
$ redis-cli dump test
"\n\x15\x15\x00\x00\x00\x12\x00\x00\x00\x05\x00\x00\xf6\x02\xf5\x02\xf4\x02\xf3\x02\xf2\xff\x06\x00\x1c\x8a\xda\x0e}\xcb\xe1."

The "printable" format cannot be used as input for the -x option which really expects the actual data. This is a misleading behavior of redis-cli.

However, there is an easy way to get the raw format:

$ redis-cli --raw dump test | hexdump -C
00000000  0a 15 15 00 00 00 12 00  00 00 05 00 00 f6 02 f5  |................|
00000010  02 f4 02 f3 02 f2 ff 06  00 1c 8a da 0e 7d cb e1  |.............}..|
00000020  2e 0a                                             |..|

Now, it is not possible to directly pipe the result of a --raw dump in a -x restore, because the last character is wrong. Compare the output of the --raw and printable dump. You will notice the --raw option adds an extra \n at the end. The raw option is not 100% raw ;-)

This extra character needs to be removed before the data can be processed by the -x option. Finally, the correct command (on a GNU/Linux system) to pipe the output of a dump in a restore is:

$ redis-cli --raw dump test | head -c-1 | redis-cli -x restore test1 0
OK

This is not pretty. I expect most people would rely of a perl/python/ruby script rather than the shell to do such tasks.

Collator

Thanks. as you said, these two commands are not designed to be use in command-line, but still this probably the best way to do so. – Mcwhorter

If you are on a mac you can brew install coreutils which will install gnu head for you with the name ghead. That way you can run the above command replacing head with ghead – Spatter

This issue occurs also when you dump/restore a single key. The solution is the same: use head -c -1. Thank you! – Anasarca

If you are copying from/to different databases: use -h flag to specify the host, -p for port and -n if you are using a database number other than the default 0. – Alexandriaalexandrian

if the value contains \n (carrige return) then, head and tr version script not work properly, you can use split/dd/truncate or other bin utils to deal with the last bad byte. – Fides

Answer 2

if the value contains \n (carrige return) then, head and tr version script not work properly, you can use split/dd/truncate or other bin utils to deal with the last bad byte.

Fides

Answer 3

Control the delimiter with -D. Set it to empty string for DUMP/RESTORE.

redis-cli --raw -D "" dump test | redis-cli -x restore test1 0

Unfold