Convert a String representation of a Dictionary to a dictionary

Question

How can I convert the str representation of a dictionary, such as the following string, into a dict?

s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"

I prefer not to use eval. What else can I use?

Candlepin

If you can't use Python 2.6, you can use a simple safeeval implmenentation like code.activestate.com/recipes/364469 It piggybacks on the Python compiler so you don't have to do all the gross work yourself. – Riser

Note: For those that come here with deceptively similar looking JSON data, you want to go read Parse JSON in Python instead. JSON is not the same thing as Python. If you have " double quotes around your strings you probably have JSON data. You can also look for null, true or false, Python syntax uses None, True and False. – Flyn

Answer 1

You can use the built-in ast.literal_eval:

>>> import ast
>>> ast.literal_eval("{'muffin' : 'lolz', 'foo' : 'kitty'}")
{'muffin': 'lolz', 'foo': 'kitty'}

This is safer than using eval. As its own docs say:

>>> help(ast.literal_eval)
Help on function literal_eval in module ast:

literal_eval(node_or_string)
    Safely evaluate an expression node or a string containing a Python
    expression.  The string or node provided may only consist of the following
    Python literal structures: strings, numbers, tuples, lists, dicts, booleans,
    and None.

For example:

>>> eval("shutil.rmtree('mongo')")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "/opt/Python-2.6.1/lib/python2.6/shutil.py", line 208, in rmtree
    onerror(os.listdir, path, sys.exc_info())
  File "/opt/Python-2.6.1/lib/python2.6/shutil.py", line 206, in rmtree
    names = os.listdir(path)
OSError: [Errno 2] No such file or directory: 'mongo'
>>> ast.literal_eval("shutil.rmtree('mongo')")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/Python-2.6.1/lib/python2.6/ast.py", line 68, in literal_eval
    return _convert(node_or_string)
  File "/opt/Python-2.6.1/lib/python2.6/ast.py", line 67, in _convert
    raise ValueError('malformed string')
ValueError: malformed string

Ilmenite

I should add that you need to sanitize the string for use with ast.literal_eval. (ensure quotes/double quotes in string are escaped) – Rainie

i get this error I am on python 2.6 (x86) on windows 7 x64 File "D:\Python26\lib\ast.py", line 48, in literal_eval node_or_string = parse(node_or_string, mode='eval') File "D:\Python26\lib\ast.py", line 36, in parse return compile(expr, filename, mode, PyCF_ONLY_AST) File "<unknown>", line 1 ^ SyntaxError: invalid syntax – Scampi

what about "dict(a=1)" style strings? – Ballonet

This doesn't seem to work for enum value inside a dictionary. Eg: d = "{'col': <Colors.RED: 2>, 'val': 2}" – Invert

why don't use json.dumps and json.loads insead, I found this solution more elevant thant using eval – Richardricharda

@Ballonet that's a statement, so it won't be allowed – Rosarosabel

I still don't see why it's better than eval(). Is it because it only accepts the structures given? – Frenchy

I have a naive question. Worried that I may have misunderstood "import". Responder says ast is built-in. Why do I need to import it? – Frenchy

@Richardricharda json.loads doesn't accept single-quoted strings. It also doesn't accept None, only null. – Ambidexter

That nice. My string dict actually included dict inside and <json.loads> method not working. – Minnich

Answer 2

https://docs.python.org/library/json.html

JSON can solve this problem, though its decoder wants double quotes around keys and values. If you don't mind a replace hack...

import json
s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"
json_acceptable_string = s.replace("'", "\"")
d = json.loads(json_acceptable_string)
# d = {u'muffin': u'lolz', u'foo': u'kitty'}

NOTE that if you have single quotes as a part of your keys or values this will fail due to improper character replacement. This solution is only recommended if you have a strong aversion to the eval solution.

More about json single quote: jQuery.parseJSON throws “Invalid JSON” error due to escaped single quote in JSON

Eadwina

Another problem is for "{0: 'Hello'}". – Loggia

This also fails if you have trailing commas (not JSON compliant), eg: "{'muffin' : 'lolz', 'foo' : 'kitty',}" – Cantu

Single-quoted strings, tuple literals, and trailing commas are not valid JSON. json.loads will only work on a valid JSON string. See the spec here: json.org Using json.loads is the safest solution, so use if possible. I would recommend transforming your input into valid JSON if necessary. – Helsie

Also this solution does not work if you have unicode strings – Olivenite

I had a similar question here #58561757 but without quotes – Towhee

Not working if value None (without single quotes, so can' t replace), e.g. "{'d': None}" – Terat

If your dict structure includes inside another dict etc ..... <json.load> not working. <ast.literal_eval >much usuful for general needs. – Minnich

Save my life for handling a dict posted by a Django form. In this case, the form is posted as a string . Thanks – Stomachache

After running into performance issues with ast.literal_eval, I am using this now. My source and inspiration: #66480573 – Financial

Answer 3

using json.loads:

>>> import json
>>> h = '{"foo":"bar", "foo2":"bar2"}'
>>> d = json.loads(h)
>>> d
{u'foo': u'bar', u'foo2': u'bar2'}
>>> type(d)
<type 'dict'>

Fiddling

I dont think it answers the OP's answer. How do we use json.laads to convert a string s = "{'muffin' : 'lolz', 'foo' : 'kitty'}" to dict? – Cement

why is this printing 'u' in the output?? eg - str = '{"1":"P", "2":"N", "3":"M"}' d = json.loads(str) print d output is : {u'1': u'P', u'3': u'M', u'2': u'N'} – Sibert

@technazi: json.loads(h.replace("'",'"')) – Hege

However, there are limits, e.g.: h= '{"muffin" : "lolz", "foo" : "kitty",}', also h= '{"muffin's" : "lolz", "foo" : "kitty"}', (just noticed part of the same comments in a similar answer... still leaving here for completeness...) – Hege

In my opinion, that's the shortest and easiest way... Definitely the one I personally prefer. – Schoolboy

@Schoolboy Too many exceptions, floating point values, tuples, etc. etc. – Jun

Thx, this seems to be the most concise way. Definitely using it from now on. But gotta keep track of trailing commas and shielded quotations – Londoner

This is the best way if the user knows that the string is a json. The clue to this is the fact that the format has double quotes. {"key" : "value"} – Envisage

Answer 4

To OP's example:

s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"

We can use Yaml to deal with this kind of non-standard json in string:

>>> import yaml
>>> s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"
>>> s
"{'muffin' : 'lolz', 'foo' : 'kitty'}"
>>> yaml.load(s)
{'muffin': 'lolz', 'foo': 'kitty'}

Cerumen

This will cause 'yes' and 'no' strings to be converted to True / False – Vaios

i got my value that works fine....but i get a error with it "AMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read msg.pyyaml.org/load for full details." what is it?? – Hobby

Only use this yaml parser for trusted input. Preferably use safe_load to avoid security implications. – Sexlimited

config = yaml.load(ymlfile, Loader=yaml.Loader) – Bangka

Answer 5

To summarize:

import ast, yaml, json, timeit

descs=['short string','long string']
strings=['{"809001":2,"848545":2,"565828":1}','{"2979":1,"30581":1,"7296":1,"127256":1,"18803":2,"41619":1,"41312":1,"16837":1,"7253":1,"70075":1,"3453":1,"4126":1,"23599":1,"11465":3,"19172":1,"4019":1,"4775":1,"64225":1,"3235":2,"15593":1,"7528":1,"176840":1,"40022":1,"152854":1,"9878":1,"16156":1,"6512":1,"4138":1,"11090":1,"12259":1,"4934":1,"65581":1,"9747":2,"18290":1,"107981":1,"459762":1,"23177":1,"23246":1,"3591":1,"3671":1,"5767":1,"3930":1,"89507":2,"19293":1,"92797":1,"32444":2,"70089":1,"46549":1,"30988":1,"4613":1,"14042":1,"26298":1,"222972":1,"2982":1,"3932":1,"11134":1,"3084":1,"6516":1,"486617":1,"14475":2,"2127":1,"51359":1,"2662":1,"4121":1,"53848":2,"552967":1,"204081":1,"5675":2,"32433":1,"92448":1}']
funcs=[json.loads,eval,ast.literal_eval,yaml.load]

for  desc,string in zip(descs,strings):
    print('***',desc,'***')
    print('')
    for  func in funcs:
        print(func.__module__+' '+func.__name__+':')
        %timeit func(string)        
    print('')

Results:

*** short string ***

json loads:
4.47 µs ± 33.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
builtins eval:
24.1 µs ± 163 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
ast literal_eval:
30.4 µs ± 299 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
yaml load:
504 µs ± 1.29 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

*** long string ***

json loads:
29.6 µs ± 230 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
builtins eval:
219 µs ± 3.92 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
ast literal_eval:
331 µs ± 1.89 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
yaml load:
9.02 ms ± 92.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Conclusion: prefer json.loads

Nolly

Except this won't work with his single-quoted string, which was part of his initial problem. Performance was never mentioned. – Tiltyard

+1 for the benchmarks (it helps making an informed decision), -1 for the conclusion: as mentioned many times, json fails in many cases. Should be up to the user to choose between features vs performance. – Argentina

Answer 6

If the string can always be trusted, you could use eval (or use literal_eval as suggested; it's safe no matter what the string is.) Otherwise you need a parser. A JSON parser (such as simplejson) would work if he only ever stores content that fits with the JSON scheme.

Marielamariele

Starting in 2.6, simplejson is included in the Python standard library as the json module. – Daphne

Yeah, that's a good answer, but note that officially JSON doesn't support single-quoted strings, as given in the original poster's example. – Emulsifier

Answer 7

Use json. the ast library consumes a lot of memory and and slower. I have a process that needs to read a text file of 156Mb. Ast with 5 minutes delay for the conversion dictionary json and 1 minutes using 60% less memory!

Ass

but has its limits: try converting the string "{'foo':'bar',}" – Hege

Answer 8

string = "{'server1':'value','server2':'value'}"

#Now removing { and }
s = string.replace("{" ,"")
finalstring = s.replace("}" , "")

#Splitting the string based on , we get key value pairs
list = finalstring.split(",")

dictionary ={}
for i in list:
    #Get Key Value pairs separately to store in dictionary
    keyvalue = i.split(":")

    #Replacing the single quotes in the leading.
    m= keyvalue[0].strip('\'')
    m = m.replace("\"", "")
    dictionary[m] = keyvalue[1].strip('"\'')

print dictionary

Garling

Many mistakes in this approach. What if value of a key contains { or }. What if it is nested dict. What if value contains , ?? – Heerlen

Answer 9

Optimized code of Siva Kameswara Rao Munipalle

s = s.replace("{", "").replace("}", "").split(",")
            
dictionary = {}

for i in s:
    dictionary[i.split(":")[0].strip('\'').replace("\"", "")] = i.split(":")[1].strip('"\'')
            
print(dictionary)

Threephase

Answer 10

no any libs are used (python2):

dict_format_string = "{'1':'one', '2' : 'two'}"
d = {}
elems  = filter(str.isalnum,dict_format_string.split("'"))
values = elems[1::2]
keys   = elems[0::2]
d.update(zip(keys,values))

NOTE: As it has hardcoded split("'") will work only for strings where data is "single quoted".

NOTE2: In python3 you need to wrap filter() to list() to get list.

Bartlett

elems = filter(str.isalnum,dict_format_string.split("'")) should be list(elems = filter(str.isalnum,dict_format_string.split("'"))) without converting to list it would still be 'filter' object – Randers

Answer 11

I couldn't use any of the above answers cause I had a string with the dtypes specified, so I used the json.load as follow:

string_dict = """{'contexts': array(['Programmed cell death (PCD) is the regulated death of cells within an organism.', ..., '...'], dtype=object)},..."""

# Replace array( with np.array(
string_dict = string_dict.replace("array(", "np.array(")

# Evaluate the string as python code
python_dict = ast.literal_eval(string_dict)

Johnette

Answer 12

If you have a string representation of a Python object that cannot be parsed by ast.literal_eval or json, then try asteval module. You can install it via pip first to use it: pip install asteval.

Its main advantage over the built-in ast or eval is that it can evaluate string representations of complex Python objects much easier than ast and much more safely than eval. Use it especially if your string contains math expressions such as nan, Decimal, numpy objects etc.

To use it, we must instantiate an Interpreter object and call it with the string to evaluate. In the example below, the string representation of the dictionary which is not JSON and contains NaN which cannot be converted by ast.literal_eval; however, asteval.Interpreter evaluates it correctly.

import ast
import json
from asteval import Interpreter

s = "{1: nan, 2: 3}"
ast.literal_eval(s)     # ValueError: malformed node or string
json.loads(s)           # JSONDecodeError
aeval = Interpreter()
aeval(s)                # {1: nan, 2: 3}

Some other examples where literal_eval or json.loads fails but asteval works.

1. If you have the string representation of numpy objects and if numpy is installed on your system, then it's much easier to convert to the proper object with asteval as well.

   aeval = Interpreter()
   aeval("{'a': array([ 1.,  2., nan])}")    # {'a': array([ 1.,  2., nan])}
   

2. By default, asteval evaluates numpy functions; however, if you want to make it parse some symbols in a certain way (e.g. numpy dtypes), you can define a custom symbol table and pass it to Interpreter at creation.

   Below is an example where a symbol table that defines how to evaluate numpy dtypes, decimal.Decimal type and null value is created and is passed to the Interpreter.

   from asteval import Interpreter, make_symbol_table
   from decimal import Decimal
   
   s = "{'a': array(['text'], dtype=object), 'b': Decimal('3.33'), 'c': null, 'd': array([1, 2], dtype=int8)}"
   symtable = make_symbol_table(object=object, Decimal=Decimal, null=None, int8='int8')
   aeval = Interpreter(symtable=symtable)
   d = aeval(s)
   print(d)   # {'a': array(['text'], dtype=object), 'b': Decimal('3.33'), 'c': None, 'd': array([1, 2], dtype=int8)}
   

---

Finally, keep in mind that since it evaluates any numpy/math functions (really any function that is fed into it as long as it is defined in the symtable argument), please be mindful of the input you pass into it.

Alien

Answer 13

My string didn't have quotes inside:
s = 'Date: 2022-11-29T10:57:01.024Z, Size: 910.11 KB'

My solution was to use str.split:
{k:v for k, v in map(lambda d: d.split(': '), s.split(', '))}

Calabash

@EricAya, why did you edit my question out? How can I do this without a map? – Calabash

Because the answers section is just for answers, not for questions, this is not a discussion forum. If you have a new question you need to post an actual new question. – Burwell