How to store and retrieve a dictionary with redis

Question

# I have the dictionary my_dict
my_dict = {
    'var1' : 5
    'var2' : 9
}
r = redis.StrictRedis()

How would I store my_dict and retrieve it with redis. For example, the following code does not work.

#Code that doesn't work
r.set('this_dict', my_dict)  # to store my_dict in this_dict
r.get('this_dict')  # to retrieve my_dict

Baca

Redis is used as a data lake here. If you don't need individual key:value pairs at retrieval time time, you can use Parquet in MinIO / S3, will be faster and more scalable (to petabytes rather than gigabytes). – Dupree

Answer 1

You can do it by hmset (multiple keys can be set using hmset).

hmset("RedisKey", dictionaryToSet)

import redis
conn = redis.Redis('localhost')

user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}

conn.hmset("pythonDict", user)

conn.hgetall("pythonDict")

{'Company': 'SCTL', 'Address': 'Mumbai', 'Location': 'RCP', 'Name': 'Pradeep'}

Dong

if it is nested data structure rather than simply dict, e.g containing some arrays etc. serialzie your data with json.dumps() write as string and after retrive from redis user json.loads() for deserializing it back to python data structure – Valeta

json.dumps() and json.loads() will only work if you are fine with your dictionary keys always being strings. If you aren't then you might consider using pickle. – Sites

json is not compatible with bytes so json serilization is not a global solution, e.g., if you have a dict with a bytes value this will not work. – Argentic

By way of note, the documentation for hmset does not tell you this, but it raises a DataError if you try to store an empty dict. – Enalda

@Pradeep how we can make the key dynamic. suppose the data is getting inserted every 15 minute so how i can make the key dynamic – Volumetric

@Volumetric you can generate a GUID. – Disgust

Be warned that the underlying Redis command HMSET has been deprecated (moved to a different function) in version 4.0.0 (July 2017) but redis-py still executes it here in hmset(). – Dupree

This method is deprecated. Use hset instead. But with the new function hset. Dictionary cannot be used anymore. – Cotangent

this doesn't work with native datetime objects either – Epistasis

pickle can be dangerous if mishandled. Use msgpack for better serialization of data before storing it into redis. – Isocrates

Answer 2

you can pickle your dict and save as string.

import pickle
import redis

r = redis.StrictRedis('localhost')
mydict = {1:2,2:3,3:4}
p_mydict = pickle.dumps(mydict)
r.set('mydict',p_mydict)

read_dict = r.get('mydict')
yourdict = pickle.loads(read_dict)

Izanagi

This is true, but depending on the rate of reads and writes, this may add serious overhead. Pickling is a slow operation – Argentic

Please note that if this is used with user input your server is prone to remote code exection, pickle.loads is should only be used on 100% trusted data – Fecundity

pickle can be dangerous if mishandled. Use msgpack for better serialization of data before storing it into Redis. – Isocrates

Pickling has also the important down-part that you cannot debug the stored data in Redis as they are binary. – Vinasse

In python3 + redis 3.x, i had to use pickle.dumps(mydict, protocol=0) and pickle.loads(str.encode(read_dict)) – Dragging

Answer 3

As the basic answer has already give by other people, I would like to add some to it.

Following are the commands in REDIS to perform basic operations with HashMap/Dictionary/Mapping type values.

1. HGET => Returns value for single key passed
2. HSET => set/updates value for the single key
3. HMGET => Returns value for single/multiple keys passed
4. HMSET => set/updates values for the multiple key
5. HGETALL => Returns all the (key, value) pairs in the mapping.

Following are their respective methods in redis-py library :-

1. HGET => hget
2. HSET => hset
3. HMGET => hmget
4. HMSET => hmset
5. HGETALL => hgetall

All of the above setter methods creates the mapping, if it doesn't exists. All of the above getter methods doesn't raise error/exceptions, if mapping/key in mapping doesn't exists.

Example:
=======
In [98]: import redis
In [99]: conn = redis.Redis('localhost')

In [100]: user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}

In [101]: con.hmset("pythonDict", {"Location": "Ahmedabad"})
Out[101]: True

In [102]: con.hgetall("pythonDict")
Out[102]:
{b'Address': b'Mumbai',
 b'Company': b'SCTL',
 b'Last Name': b'Rajpurohit',
 b'Location': b'Ahmedabad',
 b'Name': b'Mangu Singh'}

In [103]: con.hmset("pythonDict", {"Location": "Ahmedabad", "Company": ["A/C Pri
     ...: sm", "ECW", "Musikaar"]})
Out[103]: True

In [104]: con.hgetall("pythonDict")
Out[104]:
{b'Address': b'Mumbai',
 b'Company': b"['A/C Prism', 'ECW', 'Musikaar']",
 b'Last Name': b'Rajpurohit',
 b'Location': b'Ahmedabad',
 b'Name': b'Mangu Singh'}

In [105]: con.hget("pythonDict", "Name")
Out[105]: b'Mangu Singh'

In [106]: con.hmget("pythonDict", "Name", "Location")
Out[106]: [b'Mangu Singh', b'Ahmedabad']

I hope, it makes things more clear.

Algebraic

how you can make the key dynamic – Volumetric

Answer 4

If you want to store a python dict in redis, it is better to store it as json string.

import json
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)
mydict = { 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] }
rval = json.dumps(mydict)
r.set('key1', rval)

While retrieving de-serialize it using json.loads

data = r.get('key1')
result = json.loads(data)
arr = result['var3']

What about types (eg.bytes) that are not serialized by json functions ?

You can write encoder/decoder functions for types that cannot be serialized by json functions. eg. writing base64/ascii encoder/decoder function for byte array.

Deathly

I downvoted this because some dicts cannot be serialized to JSON, for example, dicts with a bytes value. – Argentic

You can write an encoder/decoder function (according to the requirement, eg. base64/ascii encoding) for the types that cannot be encoded/decoded by default. – Deathly

@Argentic - even if use hmset/hgetall , you might need to encode/decode types that are not supported by redis. – Deathly

Disagreeing about " ... latter operation is O(N)." N being the number of fields you have link to the key. Doing N SET/GET or 1 HGET/HSET is the same complexity. See : redis.io/commands/hmset Time-wise, HGET/HSET are atomic transaction, and so are performed faster by REDIS. You are just moving the complexity from Redis to Python Code. – Accountancy

The advantage of hmset is the possiblity to retrieve only certain sub-parts of the dict. With json we lose that, so this is as good as pickle or other thing. – Tramline

The import json is missing. – Vinasse

@Vinasse I agree - added it. – Mastin

Answer 5

HMSET is deprecated per the Redis docs. You can now use HSET with a dictionary as follows:

import redis

r = redis.Redis('localhost')
    
key = "hashexample" 
entry = { 
    "version":"1.2.3", 
    "tag":"main", 
    "status":"CREATED",  
    "timeout":"30"
}
r.hset(key, mapping=entry)

---

Caution: very unintuitively, hset won't accept a dictionary (raising an error suggesting it does not accept dictionaries, see [1]) if it is simply passed to the 2nd positional (unnamed) argument. You need to pass the dictionary to a named argument mapping=.

[1] *** redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.

Brune

Thanks! I'm trying to find the doc where all this is spelled out. Do you know where it is. For example, what are the two "Nones" for. – Oberhausen

@NealWalters: See the line on the HMSET command page - redis.io/commands/hmset for deprecation warning. – Breakthrough

@Tad Guski, how to you expire hashexample? – Donndonna

Answer 6

Another way: you can use RedisWorks library.

pip install redisworks

>>> from redisworks import Root
>>> root = Root()
>>> root.something = {1:"a", "b": {2: 2}}  # saves it as Hash type in Redis
...
>>> print(root.something)  # loads it from Redis
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2

It converts python types to Redis types and vice-versa.

>>> root.sides = [10, [1, 2]]  # saves it as list in Redis.
>>> print(root.sides)  # loads it from Redis
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>

Disclaimer: I wrote the library. Here is the code: https://github.com/seperman/redisworks

Cockcroft

By way of note, RedisWorks uses hmset under the hood if you set a variable to a dict, and thus if you do root.something = {} you will get a DataError, because hmset doesn't allow empty dictionaries. I mention this because the documentation for redis doesn't tell you this. – Enalda

Interesting. Yes it does use hmset. I will look into this. @Enalda – Cockcroft

But still, can it support bytes in dictionary ? – Accountancy

BTW, Redis.hmset() is deprecated. you should use Redis.hset() instead. – Jacklyn

Yeah I need to fix it when I get a chance... PR's are very welcome too! – Cockcroft

Answer 7

One might consider using MessagePack which is endorsed by redis.

import msgpack

data = {
    'one': 'one',
    'two': 2,
    'three': [1, 2, 3]
}

await redis.set('my-key', msgpack.packb(data))
val = await redis.get('my-key')
print(msgpack.unpackb(val))

# {'one': 'one', 'two': 2, 'three': [1, 2, 3]}

Using msgpack-python and aioredis

Limber

Answer 8

The redis SET command stores a string, not arbitrary data. You could try using the redis HSET command to store the dict as a redis hash with something like

for k,v in my_dict.iteritems():
    r.hset('my_dict', k, v)

but the redis datatypes and python datatypes don't quite line up. Python dicts can be arbitrarily nested, but a redis hash is going to require that your value is a string. Another approach you can take is to convert your python data to string and store that in redis, something like

r.set('this_dict', str(my_dict))

and then when you get the string out you will need to parse it to recreate the python object.

Sufferable

he can convert his data to json and store the result in redis – Siltstone

Answer 9

An other way you can approach the matter:

import redis
conn = redis.Redis('localhost')

v={'class':'user','grants': 0, 'nome': 'Roberto', 'cognome': 'Brunialti'}

y=str(v)
print(y['nome']) #<=== this return an error as y is actually a string
conn.set('test',y)

z=eval(conn.get('test'))
print(z['nome']) #<=== this really works!

I did not test it for efficiency/speed.

Cofield

Answer 10

DeprecationWarning: Redis.hmset() is deprecated. Use Redis.hset() instead.

Since HMSET is deprecated you can use HSET:

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)
r.hset('user:23', mapping={'id': 23, 'name': 'ip'})
r.hgetall('user:23')

Scuff

Answer 11

If you don't know exactly how to organize data in Redis, I did some performance tests, including the results parsing. The dictonary I used (d) had 437.084 keys (md5 format), and the values of this form:

{"path": "G:\tests\2687.3575.json",
 "info": {"f": "foo", "b": "bar"},
 "score": 2.5}

First Test (inserting data into a redis key-value mapping):

conn.hmset('my_dict', d)  # 437.084 keys added in 8.98s

conn.info()['used_memory_human']  # 166.94 Mb

for key in d:
    json.loads(conn.hget('my_dict', key).decode('utf-8').replace("'", '"'))
    #  41.1 s

import ast
for key in d:
    ast.literal_eval(conn.hget('my_dict', key).decode('utf-8'))
    #  1min 3s

conn.delete('my_dict')  # 526 ms

Second Test (inserting data directly into Redis keys):

for key in d:
    conn.hmset(key, d[key])  # 437.084 keys added in 1min 20s

conn.info()['used_memory_human']  # 326.22 Mb

for key in d:
    json.loads(conn.hgetall(key)[b'info'].decode('utf-8').replace("'", '"'))
    #  1min 11s

for key in d:
    conn.delete(key)
    #  37.3s

As you can see, in the second test, only 'info' values have to be parsed, because the hgetall(key) already returns a dict, but not a nested one.

And of course, the best example of using Redis as python's dicts, is the First Test

Barbule

Answer 12

Try rejson-py which is relatively new since 2017. Look at this introduction.

from rejson import Client, Path

rj = Client(host='localhost', port=6379)

# Set the key `obj` to some object
obj = {
    'answer': 42,
    'arr': [None, True, 3.14],
    'truth': {
        'coord': 'out there'
    }
}
rj.jsonset('obj', Path.rootPath(), obj)

# Get something
print 'Is there anybody... {}?'.format(
    rj.jsonget('obj', Path('.truth.coord'))
)

# Delete something (or perhaps nothing), append something and pop it
rj.jsondel('obj', Path('.arr[0]'))
rj.jsonarrappend('obj', Path('.arr'), 'something')
print '{} popped!'.format(rj.jsonarrpop('obj', Path('.arr')))

# Update something else
rj.jsonset('obj', Path('.answer'), 2.17)

Anthesis

Answer 13

Try the JSON format for a python dict, It is the standard according to the Redis documentation and fairly simple.

import redis

data = {
    'dog': {
        'scientific-name' : 'Canis familiaris'
    }
}

r = redis.Redis()
r.json().set('doc', '$', data)
doc = r.json().get('doc', '$')
dog = r.json().get('doc', '$.dog')
scientific_name = r.json().get('doc', '$..scientific-name')

Shirashirah

Answer 14

1. convert the dictionary to string (str)
2. Deserialize the dictionary (eval)

import redis

r = redis.Redis(port=6310, decode_responses=True)
mydict = str({ 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] })

r.set('key1', mydict)

value = eval(r.get('key1'))

Hope it help. This is a python code

Xanthene

Answer 15

Lots of good answers but this worked for me.

• store dictionary
• get dictionary
• nested hash instead of mapping the dict as key to field and value to value like other answers above. (see example 1)
• get all field/values and go from there as normally you would in a project where you want to dump a dict to a redis hash where the dict is a nested hash. (see example 2)

note: these commands were done in the python repl

1. if you want

{'field1': 'Hello', 'field2': 'World'}

Use

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
pdict = {'field1': 'Hello', 'field2': 'World'}
r.hmset("queues_test", pdict)

Also refer to other answers, particularly Saji Xavier's since its simple and works.

If you want a nested hash like

{'queue1': '{"field1": "Hello", "field2": "World"}'

then

# to set
import json
r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
pdict = {'field1': 'Hello', 'field2': 'World'}
pdict_string = json.dumps(pdict)
r.hset("queues_data", "queue1", pdict_string)

# to get a single field value
r.hget("queues_data", "queue1")
# '{"field1": "Hello", "field2": "World"}'

# to get all fields
data =  r.hgetall("queues_data")
# {'queue1': '{"field1": "Hello", "field2": "World"}'
queue1 = data['queue1']
queue1
# '{"field1": "Hello", "field2": "World"}'

result = json.loads(queue1)
result
# {'field1': 'Hello', 'field2': 'World'}
result['field1']
# 'Hello'

Then if you just need the keys/values

list(data.keys())
# ['queue1', 'queue2']

list(data.values())
# ['{"field1": "Hello", "field2": "World"}', '{"field1": "Hello", "field2": "World"}']

Then if you want get the dict back for all values in one line use

lvalues = list(data.values()) 
# ['{"field1": "Hello", "field2": "World"}', '{"field1": "Hello", "field2": "World"}']

[json.loads(x) for x in lvalues]
# [{'field1': 'Hello', 'field2': 'World'}, {'field1': 'Hello', 'field2': 'World'}]

Havstad