Copy all keys from one db to another in redis

Question

Instade of move I want to copy all my keys from a particular db to another. Is it possible in redis if yes than how ?

Betancourt

Answer 1

If you can't use MIGRATE COPY because of your redis version (2.6) you might want to copy each key separately which takes longer but doesn't require you to login to the machines themselves and allows you to move data from one database to another. Here's how I copy all keys from one database to another (but without preserving ttls)

#set connection data accordingly
source_host=localhost
source_port=6379
source_db=0
target_host=localhost
target_port=6379
target_db=1

#copy all keys without preserving ttl!
redis-cli -h $source_host -p $source_port -n $source_db keys \* | while read key; do
    echo "Copying $key"
    redis-cli --raw -h $source_host -p $source_port -n $source_db DUMP "$key" \
        | head -c -1 \
        | redis-cli -x -h $target_host -p $target_port -n $target_db RESTORE "$key" 0
done

Keys are not going to be overwritten, in order to do that, delete those keys before copying or simply flush the whole target database before starting.

Weasand

For redis versions > 2.6, you can use migrate command redis-cli keys \* | while read key; do echo "Copying $key"; abc="MIGRATE localhost 1234 $key 0 5000 COPY"; redis-cli "MIGRATE localhost 1234 $key 0 5000 COPY"; done – Aster

I've created a gist which supports also authentication and TTL. @uditmittal that's great, anyway consider that MIGRATE doesn't support authentication not even in 3.2 version. – Innocuous

In many cases, you'll want to have target_db and source_db parameters be the same (e.g. 0). 0 is the default db – Russell

Using KEYS command has serious performance implications (see 'warning' at redis.io/commands/KEYS). Consider using SCAN command instead for finding all keys to copy. – Emrich

in case of a migration on AWS ElastiCache you hate to use @Estani solution! AWS blocks some commands as the MIGRATE one : docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/… – Koziol

It's worth noting that this doesn't preserve TTL for copied keys - it'll basically persist everything in cache forever hence the "0" in the restore command. You'll need a more advanced command using PTTL, or just use MIGRATE. – Thracian

@GaryGreen indeed that's why it's mentioned in the text and code. Probably a rare case though, since not migrating anything with a TTL is like letting the entries die, which might be a valid scenario. – Weasand

@uditmittal isn't COPY only for 3.0 onwards? From redis docs: COPY and REPLACE are available only in 3.0 and above. KEYS is available starting with Redis 3.0.6. AUTH is available starting with Redis 4.0.7. – Opportunism

COPY is only available for redis 6.2.0 and above. – Morphology

Answer 2

Latest solution:

Use the RIOT open-source command line tool provided by Redislabs to copy the data.

Reference: https://developer.redis.com/riot/#_replicate_dump_and_restore

GitHub project link: https://github.com/redis-developer/riot

How to install: https://developer.redis.com/riot/#_install

# Source Redis db
SH=test1-redis.com
SP=6379

# Target Redis db
TH=test1-redis.com
TP=6379 

# Copy from db0 to db1  (standalone Redis db, Or cluster mode disabled)
# 
riot-redis -h $SH -p $SP --db 0 replicate -h $TH -p $TP --db 1 --batch 10000 \
--scan-count 10000 \
--threads 4 \
--reader-threads 4 \
--reader-batch 500 \
--reader-queue 2000 \
--reader-pool 4 

RIOT is quicker, supports multithreading, and works well with cross-environment Redis data copy ( AWS Elasticache, Redis OSS, and Redislabs ).

Underclassman

@mmoya appreciate the edits :-) – Underclassman

Answer 3

Copies all keys from database number 0 to database number 1 on localhost.

redis-cli --scan | xargs redis-cli migrate localhost 6379 '' 1 0 copy keys

If you use the same server/port you will get a timeout error but the keys seem to copy successfully anyway. GitHub Redis issue #1903

Willette

Answer 4

redis-cli -a $source_password -p $source_port -h $source_ip keys /*| while read key; 
do echo "Copying $key"; 
redis-cli --raw -a $source_password -h $source_ip -p $source_port -n $dbname DUMP "$key"| head -c -1| redis-cli -x -a $destination_password -h $destination_IP -p $destination_port RESTORE "$key" 0;

Willy

Answer 5

Not directly. I would suggest to use the always convenient redis-rdb-tools package (from Sripathi Krishnan) to extract the data from a normal rdb dump, and reinject it to another instance.

See https://github.com/sripathikrishnan/redis-rdb-tools

Sailfish

In addition to Didier's answer: if you need to do this online and incrementally, you can use a combination of SCAN and MIGRATE – Ascendancy

Answer 6

As far as I understand you need to copy keys from a particular DB (e.g 5 ) to a particular DB say 10. If that is the case you can use redis database dumper (https://github.com/r043v/rdd). Although as per documentation it has a switch (-d) to select a database for operation but didn't work for me, so what I did

1.) Edit the rdd.c file and look for int main(int argc,char argv) function
2.) Change the DB to as per your requirement
3.) compile the src by **make
4.) Dump all keys using ./rdd -o "save.rdd"
5.) Edit the rdd.c file again and change the DB
6.) Make again
7.) Import by using ./rdd "save.rdd" -o insert -s "IP" -p"Port"

Pollard

Answer 7

I know this is old, but for those of you coming here form Google:

I just published a command line interface utility to npm and github that allows you to copy keys that match a given pattern (even *) from one Redis database to another.

You can find the utility here:

https://www.npmjs.com/package/redis-utils-cli

Throughput

Answer 8

Try using dump to first dump all the keys and then restore the same

Burmese

I try this but fail to restore in the require db.Its show the error-Target key name is busy. – Betancourt

This may be because that key name already exists in target db. Try creating a fresh db and then restoring the dumped keys. Also, make sure the db is switched properly and is not the previous one while restoring which may result in conflict. – Burmese

Answer 9

If migrating keys inside of the same redis engine, then you might use internal command MOVE for that (pipelining for more speed):

#!/bin/bash

#set connection data accordingly
source_host=localhost
source_port=6379
source_db=4
target_db=0

total=$(redis-cli -n 4 keys \* | sed 's/^/MOVE /g' | sed 's/$/ '$target_db'/g' | wc -c)
#copy all keys without preserving ttl!
time redis-cli -h $source_host -p $source_port -n $source_db keys \* | \
  sed 's/^/MOVE /g' | sed 's/$/ 0/g' | \
  pv -s $total | \
  redis-cli -h $source_host -p $source_port -n $source_db >/dev/null

Corron