Encoding nested python object in JSON

Question

I want to encode objects in JSON. But, I can not figure out how to make the output without the string escaping.

import json

class Abc:
    def __init__(self):
        self.name="abc name"
    def toJSON(self):
        return json.dumps(self.__dict__, cls=ComplexEncoder)

class Doc:
    def __init__(self):
        self.abc=Abc()
    def toJSON(self):
        return json.dumps(self.__dict__, cls=ComplexEncoder)

class ComplexEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Abc) or isinstance(obj, Doc):
            return obj.toJSON()
        else:
            return json.JSONEncoder.default(self, obj)

doc=Doc()
print doc.toJSON()

The result is (the dumps returns a string representation, that's why the " are escaped)

{"abc": "{\"name\": \"abc name\"}"}

I want something a little bit different. The expected result is

{"abc": {"name": "abc name"}"}

But I don't see how to... Any hint ?

thanks in advance.

Yankee

see https://mcmap.net/q/41907/-how-to-make-a-class-json-serializable for an answer to the more general question – Phalange

Answer 1

my previous sample, with another nested object and your advices :

import json

class Identity:
    def __init__(self):
        self.name="abc name"
        self.first="abc first"
        self.addr=Addr()
    def reprJSON(self):
        return dict(name=self.name, firstname=self.first, address=self.addr) 

class Addr:
    def __init__(self):
        self.street="sesame street"
        self.zip="13000"
    def reprJSON(self):
        return dict(street=self.street, zip=self.zip) 

class Doc:
    def __init__(self):
        self.identity=Identity()
        self.data="all data"
    def reprJSON(self):
        return dict(id=self.identity, data=self.data) 

class ComplexEncoder(json.JSONEncoder):
    def default(self, obj):
        if hasattr(obj,'reprJSON'):
            return obj.reprJSON()
        else:
            return json.JSONEncoder.default(self, obj)

doc=Doc()
print "Str representation"
print doc.reprJSON()
print "Full JSON"
print json.dumps(doc.reprJSON(), cls=ComplexEncoder)
print "Partial JSON"
print json.dumps(doc.identity.addr.reprJSON(), cls=ComplexEncoder)

produces the expected result :

Str representation
{'data': 'all data', 'id': <__main__.Identity instance at 0x1005317e8>}
Full JSON
{"data": "all data", "id": {"name": "abc name", "firstname": "abc first", "address": {"street": "sesame street", "zip": "13000"}}}
Partial JSON
{"street": "sesame street", "zip": "13000"}

Thanks.

Yankee

It's been almost 9 years and your answer is still working great with Python 3.8, nice :) Thanks! – Radiolarian

Answer 2

So, the immediate problem is that you're passing the json module a JSON value, which will get encoded as just another string in the JSON value.

The broader problem is that you're greatly overcomplicating this.

Drawing on JSON datetime between Python and JavaScript, I'd go with something closer to this:

import json

class Abc:
    def __init__(self):
        self.name="abc name"
    def jsonable(self):
        return self.name

class Doc:
    def __init__(self):
        self.abc=Abc()
    def jsonable(self):
        return self.__dict__

def ComplexHandler(Obj):
    if hasattr(Obj, 'jsonable'):
        return Obj.jsonable()
    else:
        raise TypeError, 'Object of type %s with value of %s is not JSON serializable' % (type(Obj), repr(Obj))

doc=Doc()
print json.dumps(doc, default=ComplexHandler)

which gets you:

~$ python nestjson.py 
{"abc": "abc name"}
~$ 

This can be made cleaner/saner/safer (in particular, just grabbing __dict__ isn't generally a recommended thing to do outside debugging/troubleshooting), but it should get the point across. All you need, fundamentally, is a way to get a json-compatible object (whether that's a simple string or number, or a list or dict) out of each "node" in the tree. That object should not be an already-JSON-serialized object, which is what you were doing.

Calendula

Answer 3

To avoid repetition of code like in Fred Laurent's answer I overloaded the __iter__() method as follows. This also permits to 'jsonize' list elements, datetime and decimal with no extra dependencies, just use dict().

import datetime
import decimal


class Jsonable(object):
    def __iter__(self):
        for attr, value in self.__dict__.iteritems():
            if isinstance(value, datetime.datetime):
                iso = value.isoformat()
                yield attr, iso
            elif isinstance(value, decimal.Decimal):
                yield attr, str(value)
            elif(hasattr(value, '__iter__')):
                if(hasattr(value, 'pop')):
                    a = []
                    for subval in value:
                        if(hasattr(subval, '__iter__')):
                            a.append(dict(subval))
                        else:
                            a.append(subval)
                    yield attr, a
                else:
                    yield attr, dict(value)
            else:
                yield attr, value

class Identity(Jsonable):
    def __init__(self):
        self.name="abc name"
        self.first="abc first"
        self.addr=Addr()

class Addr(Jsonable):
    def __init__(self):
        self.street="sesame street"
        self.zip="13000"

class Doc(Jsonable):
    def __init__(self):
        self.identity=Identity()
        self.data="all data"


def main():
    doc=Doc()
    print "-Dictionary- \n"
    print dict(doc)
    print "\n-JSON- \n"
    print json.dumps(dict(doc), sort_keys=True, indent=4)

if __name__ == '__main__':
    main()

The output:

-Dictionary- 

{'data': 'all data', 'identity': {'first': 'abc first', 'addr': {'street': 'sesame street', 'zip': '13000'}, 'name': 'abc name'}}

-JSON- 

{
    "data": "all data", 
    "identity": {
        "addr": {
            "street": "sesame street", 
            "zip": "13000"
        }, 
        "first": "abc first", 
        "name": "abc name"
    }
}

Hope it helps! Thanks

Sarmiento

this is a brilliant solution. My question is: how might you do the same thing, but limit the contents that can be jsonized (e.g. how might I omit the name attribute from the Identity object?) – Versieversification

Answer 4

Although all the other solutions I assume they work I find they do have a lot of boilerplate code, when the goal is to only encode nested python objects.

In an article I found an elegant solution, which does exactly what you asked for but without the boilerplate code. As you can even have the de-serialization part for free as well I will show you first a solution to your exact question and then give a cleaner version where the de-serialization will work as well.

Exact solution to your question

import json


class Abc(object):
    def __init__(self):
        self.name = "abc name"


class Doc(object):
    def __init__(self):
        self.abc = Abc()


doc = Doc()

# Serialization
json_data = json.dumps(doc, default=lambda o: o.__dict__)
print(json_data)

This will output exactly what you where asking for:

{"abc": {"name": "abc name"}}

More elegant solution to enable serializing and de-seralizing

import json


class Abc(object):
    def __init__(self, name: str):
        self.name = name


class Doc(object):
    def __init__(self, abc):
        self.abc = abc


abc = Abc("abc name")
doc = Doc(abc)

# Serialization
json_data = json.dumps(doc, default=lambda o: o.__dict__)
print(json_data)

# De-serialization
decoded_doc = Doc(**json.loads(json_data))
print(decoded_doc)
print(vars(decoded_doc))

This will output the following:

{"abc": {"name": "abc name"}}
<__main__.Doc object at 0x7ff75366f250>
{'abc': {'name': 'abc name'}}

The whole magic works by defining a default lambda function: json_data = json.dumps(doc, default=lambda o: o.__dict__).

Add

this doesn't work when the are iterable attributes, like a set. – Omalley

@Omalley Indeed this does not work with set, as there does not exist as built in representation in json. Therefore it is not handled by the default JsonEncoder and JsonDecoder. In case you have data stored as a set but do not need this representation the simplest solution is to store the data as a list via list(your_data). However, if the data structure set` is needed you can find more in this link. – Add

I ran your code and decoded_doc is a Doc as desired but decoded_doc.abc is a dict and not an Abc, so this doesn't really work, does it? – Blayze

@Blayze Indeed you are right. Thank you for pointing it out. I will check in the upcoming days if there is still a solution. In case you already would have one, just shoot. – Add

TYI. Meanwhile in the source link I posted in my original answer, someone pointed this one out as well in the comments which proposed possible solutions. When I have time I will check it out. – Add

Answer 5

I could not add this as a comment and adding as answer. Fred's final sample was useful for me.I was told jsonpickle does this, but could not get the module to install and run properly. So used the code here. Minor tweak though, I had way too many variables to add by hand to some of the objects. So this little loop simplified things:

def reprJSON(self):
    d = dict()
    for a, v in self.__dict__.items():
        if (hasattr(v, "reprJSON")):
            d[a] = v.reprJSON()
        else:
            d[a] = v
    return d

It can be used in any object that has a subclass that is too busy to hand encode. Or can be made a helper for all classes. This also works for the full JSON presentation of member arrays that contain other classes (as long as they implement reprJSON() of course).

Quirinus

This was fantastic and allowed me to make it part of a class and handle attributes like datetime objects differently. Plus then the inheriting classes can call the super definition and apply their own processing to certain attributes. – Drennan

see https://mcmap.net/q/41907/-how-to-make-a-class-json-serializable for an extended version – Phalange

Answer 6

For more complex serialization I would use jsons, it was published in 2022.

• Turn Python objects into dicts or (JSON)strings and back

• No changes are required to your objects

• Easily customizable and extendable

• Works with data classes, attrs, and POPOs

    pip install jsons
    class Person:
          name:str
          birthday:datetime
  
    personObject = Person("Tony", date_of_birth)
  
    import jsons
    json_data = jsons.dumps(personObject, indent=4)
  

Frager

jsons is notoriously slow.. – Basalt

Answer 7

This is what you're looking for: https://github.com/jsonpickle/jsonpickle

It does nested serialization of Python objects and can easily be extended to serialize custom types.

Cosy

It doesnt actually – Onagraceous