How to convert JSON data into a Python object?
Asked Answered
E

32

506

I want to convert JSON data into a Python object.

I receive JSON data objects from the Facebook API, which I want to store in my database.

My current View in Django (Python) (request.POST contains the JSON):

response = request.POST
user = FbApiUser(user_id = response['id'])
user.name = response['name']
user.username = response['username']
user.save()
  • This works fine, but how do I handle complex JSON data objects?
  • Wouldn't it be much better if I could somehow convert this JSON object into a Python object for easy use?
Edmee answered 5/7, 2011 at 7:1 Comment(5)
Typically JSON gets converted to vanilla lists or dicts. Is that what you want? Or are you hoping to convert JSON straight to a custom type?Pretermit
I want to convert it into an object, something I can access using the "." . Like from the above example -> reponse.name, response.education.id etc....Edmee
Using dicts is a weak-sauce way to do object-oriented programming. Dictionaries are a very poor way to communicate expectations to readers of your code. Using a dictionary, how can you clearly and reusably specify that some dictionary keys-value pairs are required, while others aren't? What about confirming that a given value is in the acceptable range or set? What about functions that are specific to the type of object you are working with (aka methods)? Dictionaries are handy and versatile, but too many devs act like they forgot Python is an object oriented language for a reason.Colourable
There is a python library for this github.com/jsonpickle/jsonpickle (commenting since answer is too below in the thread and wont be reachable.)Standoffish
I don't think anybody has forgotten that Python is object-oriented but that would be a wonderful development. There is a reason why so many functional languages are vastly superior to OOP in so many cases. If you can manage to use stateless functions and immutable data structures you'll be better off. OOP never was the be-all-and-end-all of computing languages.Flo
G
693

UPDATE

With Python3, you can do it in one line, using SimpleNamespace and object_hook:

import json
from types import SimpleNamespace

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

# Parse JSON into an object with attributes corresponding to dict keys.
x = json.loads(data, object_hook=lambda d: SimpleNamespace(**d))
print(x.name, x.hometown.name, x.hometown.id)

OLD ANSWER (Python2)

In Python2, you can do it in one line, using namedtuple and object_hook (but it's very slow with many nested objects):

import json
from collections import namedtuple

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

# Parse JSON into an object with attributes corresponding to dict keys.
x = json.loads(data, object_hook=lambda d: namedtuple('X', d.keys())(*d.values()))
print x.name, x.hometown.name, x.hometown.id

or, to reuse this easily:

def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values())
def json2obj(data): return json.loads(data, object_hook=_json_object_hook)

x = json2obj(data)

If you want it to handle keys that aren't good attribute names, check out namedtuple's rename parameter.

Gymnasium answered 8/4, 2013 at 14:40 Comment(23)
this may result in a Value error, ValueError: Type names and field names cannot start with a number: '123'Bassorilievo
As a newbie to Python, I'm interested if this is a save thing also when security is an issue.Bobbe
It's easy for the user to supply data that will cause this to raise an exception (whether as invalid json, or an invalid attribute name -- see the note at the end about rename parameter). But other than the possibility of an exception, I don't see any potential attacks here.Gymnasium
You will find that for unwrapping data from Parse.com, rename = true will be required as DS hinted if you store complex objects. def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values(),rename=true)Bono
This turned out to be perfectly cromulent for my tiny, simple, mocking use-case. Thankies!Dextro
This creates a new different class each time encountering a JSON object while parsing, right?Lucre
Interesting. I thought relying on d.keys() and d.values() iterating in the same order is not guaranteed, but I was wrong. The docs say: "If keys, values and items views are iterated over with no intervening modifications to the dictionary, the order of items will directly correspond.". Good to know for such small, local code blocks. I'd add a comment though to explicitly alert maintainers of code of such a dependency.Sideward
Concerning the snippet that @JitendraKulkarni posted above: The rename=True argument has to supplied to the namedtuple constructor, i.e., def _json_object_hook(d): return namedtuple('X', d.keys(),rename=True)(*d.values()) .Sneakers
Is reverse operation possible?Wuhan
I am not aware of any nice general-purpose reverse operation. Any individual namedtuple can be turned to a dict using x._asdict(), which might help for simple cases.Gymnasium
It takes lot of time its good if you directly can use jsonarray or dictFrankly
For loading generic data into typed python datatypes I wrote a module called typedload. It supports nesting, lists, sets, unions and many other things.Fontanez
It returns named tuple not custom class instance.Jackofalltrades
How could this be done if our data was already a dict, not str? I mean without needing to call json.loads.Ranna
Without json.loads, I don't know of an easy way to convert recursively. To convert one level, you can use namedtuple('X', data.keys())(*data.values())Gymnasium
@BFaley perhaps just a json.dumps()?Pestalozzi
Creating a new class for every object you deserialize is absurdly inefficient. Each namedtuple class takes a ton of extra time and memory to create. This can slow down deserialization by multiple orders of magnitude.Lipoma
Literally, I tried it, and I observed over a 100x performance penalty in timing tests. The output also consumes over an order of magnitude more memory, though this is harder to measure. (It's not as simple as calling sys.getsizeof - that doesn't do what people usually think.)Lipoma
That's a good example, it is indeed pretty terrible with many objects. I added an update since Python3 provides a similar utility (SimpleNamespace) that makes this much more efficient.Gymnasium
@Gymnasium What's the reverse? Any oneliner to reserialize SimpleNamespace to JSON?Barnacle
For the reverse, you need more than a one-liner, but in a comment, I'm forced to one line... E = type("X", (json.JSONEncoder,), {"default": lambda self, o: o.__dict__ if isinstance(o, SimpleNamespace) else json.JSONEncoder.default(self, o)}), then E().encode(x)Gymnasium
My linter in vscode cannot detect type data of x.data and x.hometownNyeman
@Gymnasium irrelevant to this question, but does there exist a similar argument to object_hook=lambda d: SimpleNamespace(**d) in toml.load[s] function, I have read on the _dict argument, but could not produce a working example.Delafuente
H
187

You could try this:

class User():
    def __init__(self, name, username):
        self.name = name
        self.username = username

import json
j = json.loads(your_json)
u = User(**j)

Just create a new object and pass the parameters as a map.


You can have a JSON with objects too:

import json
class Address():
    def __init__(self, street, number):
        self.street = street
        self.number = number

    def __str__(self):
        return "{0} {1}".format(self.street, self.number)

class User():
    def __init__(self, name, address):
        self.name = name
        self.address = Address(**address)

    def __str__(self):
        return "{0} ,{1}".format(self.name, self.address)

if __name__ == '__main__':
    js = '''{"name":"Cristian", "address":{"street":"Sesame","number":122}}'''
    j = json.loads(js)
    print(j)
    u = User(**j)
    print(u)
Hodden answered 5/2, 2015 at 19:26 Comment(10)
I get TypeError: 'User' object is not subscriptableRomanticism
This should be the accepted answer. worked for me ad much simplest than all the rest.Madalene
I did not use *args, **kwargs, but the solution worked.Capitulary
User(**j) says it’s missing the name and username parameters, also how does the dict get initialized?Galliot
Works beautifully. Minimal and unobtrusive modification of original init header and simple import dictionary or json into object. Just great!Allout
what if the object User has an object?Imitative
Simplest and most effective answerPythagoreanism
Why self.address = Address(**address) instead of self.address = address ?Sackbut
@MartinThøgersen It is an example using nested objects; technically, it could be more complex than an address.Hodden
@Hodden There is no need to inherit from 'object' classJacobi
P
146

Check out the section titled Specializing JSON object decoding in the json module documentation. You can use that to decode a JSON object into a specific Python type.

Here's an example:

class User(object):
    def __init__(self, name, username):
        self.name = name
        self.username = username

import json
def object_decoder(obj):
    if '__type__' in obj and obj['__type__'] == 'User':
        return User(obj['name'], obj['username'])
    return obj

json.loads('{"__type__": "User", "name": "John Smith", "username": "jsmith"}',
           object_hook=object_decoder)

print type(User)  # -> <type 'type'>

Update

If you want to access data in a dictionary via the json module do this:

user = json.loads('{"__type__": "User", "name": "John Smith", "username": "jsmith"}')
print user['name']
print user['username']

Just like a regular dictionary.

Pretermit answered 5/7, 2011 at 7:19 Comment(6)
Hey, I was just reading up and I realized that dictionaries will totally do, only I was wondering how to convert JSON objects into dictionaries and how do I access this data from the dictionary?Edmee
Awesome, it's almost clear, just wanted to know one more little thing that if there's this object -> { 'education' : { 'name1' : 456 , 'name2' : 567 } }, how do i access this data?Edmee
it'd just be topLevelData['education']['name1'] ==> 456. make sense?Pretermit
@Ben: I think your comment is inappropriate. Of all the answers here currently it is the only one to get the classes right. Which means: It's a one-pass operation and the result uses the correct types. Pickle itself is for different applications than JSON (binary versus textual rep) and jsonpickle is a nonstandard lib. I'd be interested to see how you solve the issue that the std json lib does not provide the upper parse tree to the object hookSideward
I have to agree with @Ben on this. This is a really bad solution. Not scalable at all. You'll need to maintain fields' names as string and as field. If you'll want to refactor your fields the decoding will fail (of course the already serialized data will no longer be relevant anyway). The same concept is already implemented well with jsonpickleAlanson
This is quite messy and complicated and I can't recommend this approach in 2020.Perturb
W
113

This is not code golf, but here is my shortest trick, using types.SimpleNamespace as the container for JSON objects.

Compared to the leading namedtuple solution, it is:

  • probably faster/smaller as it does not create a class for each object
  • shorter
  • no rename option, and probably the same limitation on keys that are not valid identifiers (uses setattr under the covers)

Example:

from __future__ import print_function
import json

try:
    from types import SimpleNamespace as Namespace
except ImportError:
    # Python 2.x fallback
    from argparse import Namespace

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

x = json.loads(data, object_hook=lambda d: Namespace(**d))

print (x.name, x.hometown.name, x.hometown.id)
Waechter answered 10/3, 2016 at 15:22 Comment(5)
By the way, the serialization library Marshmallow offers a similar feature with its @post_load decorator. marshmallow.readthedocs.io/en/latest/…Filip
To avoid the dependency on argparse: replace the argparse import with from types import SimpleNamespace and use: x = json.loads(data, object_hook=lambda d: SimpleNamespace(**d))Rankin
Edited to use @maxschlepzig's solution when running under Python 3.x (types.SimpleNamespace doesn't exist in 2.7, unfortunately).Oblivion
This is by far the cleanest approach. The only thing to be pointed out that SimpleNamespace will parse JSON-booleans "true" or "false" literally - in those cases 1s and 0s can be used in the JSON to establish truthiness instead.Tipster
@VineetBansal: Whatever effect you're talking about isn't something that SimpleNamespace does. Something might be wrong with your JSON.Lipoma
C
44

Here's a quick and dirty json pickle alternative

import json

class User:
    def __init__(self, name, username):
        self.name = name
        self.username = username

    def to_json(self):
        return json.dumps(self.__dict__)

    @classmethod
    def from_json(cls, json_str):
        json_dict = json.loads(json_str)
        return cls(**json_dict)

# example usage
User("tbrown", "Tom Brown").to_json()
User.from_json(User("tbrown", "Tom Brown").to_json()).to_json()
Crib answered 21/10, 2015 at 23:29 Comment(1)
This is not good approach. At first to_json and from_json should not be placed in your class. At second it will not work work for nested classes.Perturb
F
18

For complex objects, you can use JSON Pickle

Python library for serializing any arbitrary object graph into JSON. It can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python.

Ferraro answered 5/7, 2011 at 7:19 Comment(5)
I think jsonstruct is better. jsonstruct originally a fork of jsonpickle (Thanks guys!). The key difference between this library and jsonpickle is that during deserialization, jsonpickle requires Python types to be recorded as part of the JSON. This library intends to remove this requirement, instead, requires a class to be passed in as an argument so that its definition can be inspected. It will then return an instance of the given class. This approach is similar to how Jackson (of Java) works.Succory
The problems with jsonstruct is that it doesn't appear to be maintained (in fact, it looks abandoned) and it fails to convert a list of objects, like '[{"name":"object1"},{"name":"object2"}]'. jsonpickle doesn't handle it very well, either.Katherinkatherina
I have no idea why this answer isn't getting more votes. Most other solution are quite out-there. Someone developed a great library for JSON de/serialization - why not use it? In addition, seems to be working fine with lists - what was your issue with it @LS ?Alanson
@guyarad, the problem is: x= jsonpickle.decode('[{"name":"object1"},{"name":"object2"}]') gives a list of dictionaries ([{'name': 'object1'}, {'name': 'object2'}]), not a list of objects with properties (x[0].name == 'object1'), which is what the original question required. To get that, I ended up using the object_hook/Namespace approach suggested by eddygeek, but the quick/dirty approach by ubershmekel looks good, too. I think I could use object_hook with jsonpickle's set_encoder_options() (undocumented!), but it would take more code than the basic json module. I'd love to be proven wrong!Katherinkatherina
@LS if you have no control over the input, which is truly what the OP asked, jsonpickle isn't ideal since it expect the actual type in each level (and will assume basic types if missing). Both solutions are "cute".Alanson
I
17

If you're using Python 3.5+, you can use jsons to serialize and deserialize to plain old Python objects:

import jsons

response = request.POST

# You'll need your class attributes to match your dict keys, so in your case do:
response['id'] = response.pop('user_id')

# Then you can load that dict into your class:
user = jsons.load(response, FbApiUser)

user.save()

You could also make FbApiUser inherit from jsons.JsonSerializable for more elegance:

user = FbApiUser.from_json(response)

These examples will work if your class consists of Python default types, like strings, integers, lists, datetimes, etc. The jsons lib will require type hints for custom types though.

Ihab answered 17/12, 2018 at 22:13 Comment(0)
E
14

If you are using python 3.6+, you can use marshmallow-dataclass. Contrarily to all the solutions listed above, it is both simple, and type safe:

from marshmallow_dataclass import dataclass

@dataclass
class User:
    name: str

user = User.Schema().load({"name": "Ramirez"})
Echidna answered 20/2, 2019 at 17:48 Comment(2)
TypeError: make_data_class() got an unexpected keyword argument 'many'Discount
@Discount : You should open an issue with a reproducible test case in github.com/lovasoa/marshmallow_dataclass/issuesEchidna
S
9

Since no one provided an answer quite like mine, I am going to post it here.

It is a robust class that can easily convert back and forth between JSON str and dict that I have copied from my answer to another question:

import json

class PyJSON(object):
    def __init__(self, d):
        if type(d) is str:
            d = json.loads(d)

        self.from_dict(d)

    def from_dict(self, d):
        self.__dict__ = {}
        for key, value in d.items():
            if type(value) is dict:
                value = PyJSON(value)
            self.__dict__[key] = value

    def to_dict(self):
        d = {}
        for key, value in self.__dict__.items():
            if type(value) is PyJSON:
                value = value.to_dict()
            d[key] = value
        return d

    def __repr__(self):
        return str(self.to_dict())

    def __setitem__(self, key, value):
        self.__dict__[key] = value

    def __getitem__(self, key):
        return self.__dict__[key]

json_str = """... JSON string ..."""

py_json = PyJSON(json_str)
Summary answered 31/1, 2019 at 18:56 Comment(0)
A
8

Improving the lovasoa's very good answer.

If you are using python 3.6+, you can use:
pip install marshmallow-enum and
pip install marshmallow-dataclass

Its simple and type safe.

You can transform your class in a string-json and vice-versa:

From Object to String Json:

    from marshmallow_dataclass import dataclass
    user = User("Danilo","50","RedBull",15,OrderStatus.CREATED)
    user_json = User.Schema().dumps(user)
    user_json_str = user_json.data

From String Json to Object:

    json_str = '{"name":"Danilo", "orderId":"50", "productName":"RedBull", "quantity":15, "status":"Created"}'
    user, err = User.Schema().loads(json_str)
    print(user,flush=True)

Class definitions:

class OrderStatus(Enum):
    CREATED = 'Created'
    PENDING = 'Pending'
    CONFIRMED = 'Confirmed'
    FAILED = 'Failed'

@dataclass
class User:
    def __init__(self, name, orderId, productName, quantity, status):
        self.name = name
        self.orderId = orderId
        self.productName = productName
        self.quantity = quantity
        self.status = status

    name: str
    orderId: str
    productName: str
    quantity: int
    status: OrderStatus
Anhinga answered 22/5, 2019 at 19:37 Comment(1)
You dont need the constructor, just pass init=True to dataclass and you are good to go.Obey
R
8

dacite may also be a solution for you, it supports following features:

  • nested structures
  • (basic) types checking
  • optional fields (i.e. typing.Optional)
  • unions
  • forward references
  • collections
  • custom type hooks

https://pypi.org/project/dacite/

from dataclasses import dataclass
from dacite import from_dict


@dataclass
class User:
    name: str
    age: int
    is_active: bool


data = {
    'name': 'John',
    'age': 30,
    'is_active': True,
}

user = from_dict(data_class=User, data=data)

assert user == User(name='John', age=30, is_active=True)
Rena answered 2/7, 2020 at 20:24 Comment(0)
K
6

JSON to python object

The follwing code creates dynamic attributes with the objects keys recursively.

JSON object - fb_data.json:

{
    "name": "John Smith",
    "hometown": {
        "name": "New York",
        "id": 123
    },
    "list": [
        "a",
        "b",
        "c",
        1,
        {
            "key": 1
        }
    ],
    "object": {
        "key": {
            "key": 1
        }
    }
}

On the conversion we have 3 cases:

  • lists
  • dicts (new object)
  • bool, int, float and str
import json


class AppConfiguration(object):
    def __init__(self, data=None):
        if data is None:
            with open("fb_data.json") as fh:
                data = json.loads(fh.read())
        else:
            data = dict(data)

        for key, val in data.items():
            setattr(self, key, self.compute_attr_value(val))

    def compute_attr_value(self, value):
        if isinstance(value, list):
            return [self.compute_attr_value(x) for x in value]
        elif isinstance(value, dict):
            return AppConfiguration(value)
        else:
            return value


if __name__ == "__main__":
    instance = AppConfiguration()

    print(instance.name)
    print(instance.hometown.name)
    print(instance.hometown.id)
    print(instance.list[4].key)
    print(instance.object.key.key)

Now the key, value pairs are attributes - objects.

output:

John Smith
New York
123
1
1

Paste JSON as Code

Supports TypeScript, Python, Go, Ruby, C#, Java, Swift, Rust, Kotlin, C++, Flow, Objective-C, JavaScript, Elm, and JSON Schema.

  • Interactively generate types and (de-)serialization code from JSON, JSON Schema, and TypeScript
  • Paste JSON/JSON Schema/TypeScript as code

enter image description here

quicktype infers types from sample JSON data, then outputs strongly typed models and serializers for working with that data in your desired programming language.

output:

# Generated by https://quicktype.io
#
# To change quicktype's target language, run command:
#
#   "Set quicktype target language"

from typing import List, Union


class Hometown:
    name: str
    id: int

    def __init__(self, name: str, id: int) -> None:
        self.name = name
        self.id = id


class Key:
    key: int

    def __init__(self, key: int) -> None:
        self.key = key


class Object:
    key: Key

    def __init__(self, key: Key) -> None:
        self.key = key


class FbData:
    name: str
    hometown: Hometown
    list: List[Union[Key, int, str]]
    object: Object

    def __init__(self, name: str, hometown: Hometown, list: List[Union[Key, int, str]], object: Object) -> None:
        self.name = name
        self.hometown = hometown
        self.list = list
        self.object = object

This extension is available for free in the Visual Studio Code Marketplace.

Krieg answered 4/2, 2021 at 21:27 Comment(2)
Just saw that you can even use it online: quicktype.io appCusco
For single use, I guess an online solution can help. For automation of the process, ie for repeating the steps, the online solution is not usable. In that example, the written solution would be adapted to the needs in order to successfully solve the problem.Frugivorous
N
5

I have written a small (de)serialization framework called any2any that helps doing complex transformations between two Python types.

In your case, I guess you want to transform from a dictionary (obtained with json.loads) to an complex object response.education ; response.name, with a nested structure response.education.id, etc ... So that's exactly what this framework is made for. The documentation is not great yet, but by using any2any.simple.MappingToObject, you should be able to do that very easily. Please ask if you need help.

Nightstick answered 4/8, 2011 at 14:29 Comment(3)
Sebpiq, have installed any2any and am having troubles understanding the intended sequence of method calls. Could you give a simple example of converting a dictionary to a Python object with a property for each key?Mcalister
Hi @Mcalister ! If you have installed it from pypi, the version is completely out of date, I have made a complete refactoring a few weeks ago. You should use the github version (I need to make a proper release !)Nightstick
I installed it from pypy because the github said to install it from pypy. Also, you said pypy was out of date months ago.. It didn't work :( I filed a bug report tho! github.com/sebpiq/any2any/issues/11Jaxartes
B
5

The lightest solution I think is

import orjson  # faster then json =)
from typing import NamedTuple

_j = '{"name":"Иван","age":37,"mother":{"name":"Ольга","age":58},"children":["Маша","Игорь","Таня"],"married": true,' \
     '"dog":null} '


class PersonNameAge(NamedTuple):
    name: str
    age: int


class UserInfo(NamedTuple):
    name: str
    age: int
    mother: PersonNameAge
    children: list
    married: bool
    dog: str


j = orjson.loads(_j)
u = UserInfo(**j)

print(u.name, u.age, u.mother, u.children, u.married, u.dog)

>>> Ivan 37 {'name': 'Olga', 'age': 58} ['Mary', 'Igor', 'Jane'] True None
Batt answered 13/1, 2021 at 23:35 Comment(1)
Oddly, the PersonNameAge doesn't unpack itself, right? I still get a dict instead of a PersonNameAge object for mother.Remark
T
5

If you are looking for type safe deserialization of JSON or any complex dict into a python class I would highly recommend pydantic for Python 3.7+. Not only does it has a succinct API (does not require writing 'helper' boilerplate), can integrate with Python dataclasses but has static and runtime type validation of complex and nested data structures.

Example usage:

from pydantic import BaseModel
from datetime import datetime

class Item(BaseModel):
    field1: str | int           # union
    field2: int | None = None   # optional
    field3: str = 'default'     # default values

class User(BaseModel):
    name: str | None = None
    username: str
    created: datetime           # default type converters
    items: list[Item] = []      # nested complex types

data = {
    'name': 'Jane Doe',
    'username': 'user1',
    'created': '2020-12-31T23:59:00+10:00',
    'items': [
        {'field1': 1, 'field2': 2},
        {'field1': 'b'},
        {'field1': 'c', 'field3': 'override'}
    ]
}

user: User = User(**data)

For more details and features, check out pydantic's rational section in their documentation.

Tokharian answered 23/9, 2022 at 7:52 Comment(0)
G
4

Expanding on DS's answer a bit, if you need the object to be mutable (which namedtuple is not), you can use the recordclass library instead of namedtuple:

import json
from recordclass import recordclass

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

# Parse into a mutable object
x = json.loads(data, object_hook=lambda d: recordclass('X', d.keys())(*d.values()))

The modified object can then be converted back to json very easily using simplejson:

x.name = "John Doe"
new_json = simplejson.dumps(x)
Glia answered 19/1, 2019 at 15:30 Comment(0)
S
4

dataclass-wizard is a modern option that can similarly work for you. It supports auto key casing transforms, such as camelCase or TitleCase, both of which is quite common in API responses.

The default key transform when dumping instance to a dict/JSON is camelCase, but this can be easily overriden using a Meta config supplied on the main dataclass.

https://pypi.org/project/dataclass-wizard/

from dataclasses import dataclass

from dataclass_wizard import fromdict, asdict


@dataclass
class User:
    name: str
    age: int
    is_active: bool


data = {
    'name': 'John',
    'age': 30,
    'isActive': True,
}

user = fromdict(User, data)
assert user == User(name='John', age=30, is_active=True)

json_dict = asdict(user)
assert json_dict == {'name': 'John', 'age': 30, 'isActive': True}

Example of setting a Meta config, which converts fields to lisp-case when serializing to dict/JSON:

DumpMeta(key_transform='LISP').bind_to(User)
Sibley answered 9/12, 2021 at 0:46 Comment(0)
R
3

While searching for a solution, I've stumbled upon this blog post: https://blog.mosthege.net/2016/11/12/json-deserialization-of-nested-objects/

It uses the same technique as stated in previous answers but with a usage of decorators. Another thing I found useful is the fact that it returns a typed object at the end of deserialisation

class JsonConvert(object):
    class_mappings = {}

    @classmethod
    def class_mapper(cls, d):
        for keys, cls in clsself.mappings.items():
            if keys.issuperset(d.keys()):   # are all required arguments present?
                return cls(**d)
        else:
            # Raise exception instead of silently returning None
            raise ValueError('Unable to find a matching class for object: {!s}'.format(d))

    @classmethod
    def complex_handler(cls, Obj):
        if hasattr(Obj, '__dict__'):
            return Obj.__dict__
        else:
            raise TypeError('Object of type %s with value of %s is not JSON serializable' % (type(Obj), repr(Obj)))

    @classmethod
    def register(cls, claz):
        clsself.mappings[frozenset(tuple([attr for attr,val in cls().__dict__.items()]))] = cls
        return cls

    @classmethod
    def to_json(cls, obj):
        return json.dumps(obj.__dict__, default=cls.complex_handler, indent=4)

    @classmethod
    def from_json(cls, json_str):
        return json.loads(json_str, object_hook=cls.class_mapper)

Usage:

@JsonConvert.register
class Employee(object):
    def __init__(self, Name:int=None, Age:int=None):
        self.Name = Name
        self.Age = Age
        return

@JsonConvert.register
class Company(object):
    def __init__(self, Name:str="", Employees:[Employee]=None):
        self.Name = Name
        self.Employees = [] if Employees is None else Employees
        return

company = Company("Contonso")
company.Employees.append(Employee("Werner", 38))
company.Employees.append(Employee("Mary"))

as_json = JsonConvert.to_json(company)
from_json = JsonConvert.from_json(as_json)
as_json_from_json = JsonConvert.to_json(from_json)

assert(as_json_from_json == as_json)

print(as_json_from_json)
Radical answered 5/10, 2018 at 10:39 Comment(0)
D
2

The answers given here does not return the correct object type, hence I created these methods below. They also fail if you try to add more fields to the class that does not exist in the given JSON:

def dict_to_class(class_name: Any, dictionary: dict) -> Any:
    instance = class_name()
    for key in dictionary.keys():
        setattr(instance, key, dictionary[key])
    return instance


def json_to_class(class_name: Any, json_string: str) -> Any:
    dict_object = json.loads(json_string)
    return dict_to_class(class_name, dict_object)
Deutschland answered 16/3, 2020 at 16:51 Comment(0)
N
2

There are multiple viable answers already, but there are some minor libraries made by individuals that can do the trick for most users.

An example would be json2object. Given a defined class, it deserialises json data to your custom model, including custom attributes and child objects.

Its use is very simple. An example from the library wiki:

from json2object import jsontoobject as jo

class Student:
    def __init__(self):
        self.firstName = None
        self.lastName = None
        self.courses = [Course('')]

class Course:
    def __init__(self, name):
        self.name = name

data = '''{
"firstName": "James",
"lastName": "Bond",
"courses": [{
    "name": "Fighting"},
    {
    "name": "Shooting"}
    ]
}
'''

model = Student()
result = jo.deserialize(data, model)
print(result.courses[0].name)
Newcomb answered 4/11, 2020 at 19:43 Comment(1)
quicktype.io as proposed by Milovan above does a slightly better job, as it uses more features offered by Python. But sometimes it would be definitely more useful to have a python library!Cusco
M
2
class SimpleClass:
    def __init__(self, **kwargs):
        for k, v in kwargs.items():
            if type(v) is dict:
                setattr(self, k, SimpleClass(**v))
            else:
                setattr(self, k, v)


json_dict = {'name': 'jane doe', 'username': 'jane', 'test': {'foo': 1}}

class_instance = SimpleClass(**json_dict)

print(class_instance.name, class_instance.test.foo)
print(vars(class_instance))
Marilynnmarimba answered 7/8, 2021 at 23:33 Comment(0)
R
2

This appears to be an XY problem (asking A where the actual problem is B).

The root of the issue is: How to effectively reference/modify deep-nested JSON structures without having to do obj['foo']['bar'][42]['quux'], which poses a typing challenge, a code-bloat issue, a readability issue and an error-trapping issue?

Use glom

from glom import glom

# Basic deep get

data = {'a': {'b': {'c': 'd'}}}

print(glom(data, 'a.b.c'))

It will handle list items also: glom(data, 'a.b.c.42.d')

I've benchmarked it against a naive implementation:

def extract(J, levels):
    # Twice as fast as using glom
    for level in levels.split('.'):
        J = J[int(level) if level.isnumeric() else level]
    return J

... and it returns 0.14ms on a complex JSON object, compared with 0.06ms for the naive impl.

It can also handle comlex queries, e.g. pulling out all foo.bar.records where .name == 'Joe Bloggs'

EDIT:

Another performant approach is to recursively use a class that overrides __getitem__ and __getattr__:

class Ob:
    def __init__(self, J):
        self.J = J

    def __getitem__(self, index):
        return Ob(self.J[index])

    def __getattr__(self, attr):
        value = self.J.get(attr, None)
        return Ob(value) if type(value) in (list, dict) else value

Now you can do:

ob = Ob(J)

# if you're fetching a final raw value (not list/dict
ob.foo.bar[42].quux.leaf

# for intermediate values
ob.foo.bar[42].quux.J

This also benchmarks surprisingly well. Comparable with my previous naive impl. If anyone can spot a way to tidy up access for non-leaf queries, leave a comment!

Remodel answered 2/10, 2021 at 6:55 Comment(0)
O
1

Modifying @DS response a bit, to load from a file:

def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values())
def load_data(file_name):
  with open(file_name, 'r') as file_data:
    return file_data.read().replace('\n', '')
def json2obj(file_name): return json.loads(load_data(file_name), object_hook=_json_object_hook)

One thing: this cannot load items with numbers ahead. Like this:

{
  "1_first_item": {
    "A": "1",
    "B": "2"
  }
}

Because "1_first_item" is not a valid python field name.

Osmund answered 6/12, 2016 at 8:45 Comment(0)
Q
1

If you're using Python 3.6 or newer, you could have a look at squema - a lightweight module for statically typed data structures. It makes your code easy to read while at the same time providing simple data validation, conversion and serialization without extra work. You can think of it as a more sophisticated and opinionated alternative to namedtuples and dataclasses. Here's how you could use it:

from uuid import UUID
from squema import Squema


class FbApiUser(Squema):
    id: UUID
    age: int
    name: str

    def save(self):
        pass


user = FbApiUser(**json.loads(response))
user.save()
Qianaqibla answered 1/10, 2019 at 18:16 Comment(1)
This is also more similar to JVM language ways to do it.Aimless
H
1

You can use

x = Map(json.loads(response))
x.__class__ = MyClass

where

class Map(dict):
    def __init__(self, *args, **kwargs):
        super(Map, self).__init__(*args, **kwargs)
        for arg in args:
            if isinstance(arg, dict):
                for k, v in arg.iteritems():
                    self[k] = v
                    if isinstance(v, dict):
                        self[k] = Map(v)

        if kwargs:
            # for python 3 use kwargs.items()
            for k, v in kwargs.iteritems():
                self[k] = v
                if isinstance(v, dict):
                    self[k] = Map(v)

    def __getattr__(self, attr):
        return self.get(attr)

    def __setattr__(self, key, value):
        self.__setitem__(key, value)

    def __setitem__(self, key, value):
        super(Map, self).__setitem__(key, value)
        self.__dict__.update({key: value})

    def __delattr__(self, item):
        self.__delitem__(item)

    def __delitem__(self, key):
        super(Map, self).__delitem__(key)
        del self.__dict__[key]

For a generic, future-proof solution.

Hanuman answered 4/12, 2019 at 12:9 Comment(0)
N
1

I was searching for a solution that worked with recordclass.RecordClass, supports nested objects and works for both json serialization and json deserialization.

Expanding on DS's answer, and expanding on solution from BeneStr, I came up with the following that seems to work:

Code:

import json
import recordclass

class NestedRec(recordclass.RecordClass):
    a : int = 0
    b : int = 0

class ExampleRec(recordclass.RecordClass):
    x : int       = None
    y : int       = None
    nested : NestedRec = NestedRec()

class JsonSerializer:
    @staticmethod
    def dumps(obj, ensure_ascii=True, indent=None, sort_keys=False):
        return json.dumps(obj, default=JsonSerializer.__obj_to_dict, ensure_ascii=ensure_ascii, indent=indent, sort_keys=sort_keys)

    @staticmethod
    def loads(s, klass):
        return JsonSerializer.__dict_to_obj(klass, json.loads(s))

    @staticmethod
    def __obj_to_dict(obj):
        if hasattr(obj, "_asdict"):
            return obj._asdict()
        else:
            return json.JSONEncoder().default(obj)

    @staticmethod
    def __dict_to_obj(klass, s_dict):
        kwargs = {
            key : JsonSerializer.__dict_to_obj(cls, s_dict[key]) if hasattr(cls,'_asdict') else s_dict[key] \
                for key,cls in klass.__annotations__.items() \
                    if s_dict is not None and key in s_dict
        }
        return klass(**kwargs)

Usage:

example_0 = ExampleRec(x = 10, y = 20, nested = NestedRec( a = 30, b = 40 ) )

#Serialize to JSON

json_str = JsonSerializer.dumps(example_0)
print(json_str)
#{
#  "x": 10,
#  "y": 20,
#  "nested": {
#    "a": 30,
#    "b": 40
#  }
#}

# Deserialize from JSON
example_1 = JsonSerializer.loads(json_str, ExampleRec)
example_1.x += 1
example_1.y += 1
example_1.nested.a += 1
example_1.nested.b += 1

json_str = JsonSerializer.dumps(example_1)
print(json_str)
#{
#  "x": 11,
#  "y": 21,
#  "nested": {
#    "a": 31,
#    "b": 41
#  }
#}
Nieshanieto answered 11/1, 2020 at 11:9 Comment(0)
C
1
def load_model_from_dict(self, data: dict):
    for key, value in data.items():
        self.__dict__[key] = value
    return self

It help returns your own model, with unforeseenable variables from the dict.

Creatinine answered 25/7, 2022 at 7:4 Comment(0)
A
1

So I was hunting for a way to unmarshal any arbitrary type (think dict of dataclass, or dict of a dict of an array of dataclass) without a ton of custom deserialization code.

This is my approach:

import json
from dataclasses import dataclass, make_dataclass

from dataclasses_json import DataClassJsonMixin, dataclass_json


@dataclass_json
@dataclass
class Person:
    name: str


def unmarshal_json(data, t):
    Unmarhsal = make_dataclass('Unmarhsal', [('res', t)],
                               bases=(DataClassJsonMixin,))
    d = json.loads(data)
    out = Unmarhsal.from_dict({"res": d})
    return out.res


unmarshalled = unmarshal_json('{"1": {"name": "john"} }', dict[str, Person])
print(unmarshalled)

Prints: {'1': Person(name='john')}

Appetency answered 26/8, 2022 at 21:36 Comment(0)
A
0

Python3.x

The best aproach I could reach with my knowledge was this.
Note that this code treat set() too.
This approach is generic just needing the extension of class (in the second example).
Note that I'm just doing it to files, but it's easy to modify the behavior to your taste.

However this is a CoDec.

With a little more work you can construct your class in other ways. I assume a default constructor to instance it, then I update the class dict.

import json
import collections


class JsonClassSerializable(json.JSONEncoder):

    REGISTERED_CLASS = {}

    def register(ctype):
        JsonClassSerializable.REGISTERED_CLASS[ctype.__name__] = ctype

    def default(self, obj):
        if isinstance(obj, collections.Set):
            return dict(_set_object=list(obj))
        if isinstance(obj, JsonClassSerializable):
            jclass = {}
            jclass["name"] = type(obj).__name__
            jclass["dict"] = obj.__dict__
            return dict(_class_object=jclass)
        else:
            return json.JSONEncoder.default(self, obj)

    def json_to_class(self, dct):
        if '_set_object' in dct:
            return set(dct['_set_object'])
        elif '_class_object' in dct:
            cclass = dct['_class_object']
            cclass_name = cclass["name"]
            if cclass_name not in self.REGISTERED_CLASS:
                raise RuntimeError(
                    "Class {} not registered in JSON Parser"
                    .format(cclass["name"])
                )
            instance = self.REGISTERED_CLASS[cclass_name]()
            instance.__dict__ = cclass["dict"]
            return instance
        return dct

    def encode_(self, file):
        with open(file, 'w') as outfile:
            json.dump(
                self.__dict__, outfile,
                cls=JsonClassSerializable,
                indent=4,
                sort_keys=True
            )

    def decode_(self, file):
        try:
            with open(file, 'r') as infile:
                self.__dict__ = json.load(
                    infile,
                    object_hook=self.json_to_class
                )
        except FileNotFoundError:
            print("Persistence load failed "
                  "'{}' do not exists".format(file)
                  )


class C(JsonClassSerializable):

    def __init__(self):
        self.mill = "s"


JsonClassSerializable.register(C)


class B(JsonClassSerializable):

    def __init__(self):
        self.a = 1230
        self.c = C()


JsonClassSerializable.register(B)


class A(JsonClassSerializable):

    def __init__(self):
        self.a = 1
        self.b = {1, 2}
        self.c = B()

JsonClassSerializable.register(A)

A().encode_("test")
b = A()
b.decode_("test")
print(b.a)
print(b.b)
print(b.c.a)

Edit

With some more of research I found a way to generalize without the need of the SUPERCLASS register method call, using a metaclass

import json
import collections

REGISTERED_CLASS = {}

class MetaSerializable(type):

    def __call__(cls, *args, **kwargs):
        if cls.__name__ not in REGISTERED_CLASS:
            REGISTERED_CLASS[cls.__name__] = cls
        return super(MetaSerializable, cls).__call__(*args, **kwargs)


class JsonClassSerializable(json.JSONEncoder, metaclass=MetaSerializable):

    def default(self, obj):
        if isinstance(obj, collections.Set):
            return dict(_set_object=list(obj))
        if isinstance(obj, JsonClassSerializable):
            jclass = {}
            jclass["name"] = type(obj).__name__
            jclass["dict"] = obj.__dict__
            return dict(_class_object=jclass)
        else:
            return json.JSONEncoder.default(self, obj)

    def json_to_class(self, dct):
        if '_set_object' in dct:
            return set(dct['_set_object'])
        elif '_class_object' in dct:
            cclass = dct['_class_object']
            cclass_name = cclass["name"]
            if cclass_name not in REGISTERED_CLASS:
                raise RuntimeError(
                    "Class {} not registered in JSON Parser"
                    .format(cclass["name"])
                )
            instance = REGISTERED_CLASS[cclass_name]()
            instance.__dict__ = cclass["dict"]
            return instance
        return dct

    def encode_(self, file):
        with open(file, 'w') as outfile:
            json.dump(
                self.__dict__, outfile,
                cls=JsonClassSerializable,
                indent=4,
                sort_keys=True
            )

    def decode_(self, file):
        try:
            with open(file, 'r') as infile:
                self.__dict__ = json.load(
                    infile,
                    object_hook=self.json_to_class
                )
        except FileNotFoundError:
            print("Persistence load failed "
                  "'{}' do not exists".format(file)
                  )


class C(JsonClassSerializable):

    def __init__(self):
        self.mill = "s"


class B(JsonClassSerializable):

    def __init__(self):
        self.a = 1230
        self.c = C()


class A(JsonClassSerializable):

    def __init__(self):
        self.a = 1
        self.b = {1, 2}
        self.c = B()


A().encode_("test")
b = A()
b.decode_("test")
print(b.a)
# 1
print(b.b)
# {1, 2}
print(b.c.a)
# 1230
print(b.c.c.mill)
# s
Affra answered 19/8, 2018 at 4:47 Comment(0)
F
0

this is not a very difficult thing, i saw the answers above, most of them had a performance problem in the "list"

this code is much faster than the above

import json 

class jsonify:
    def __init__(self, data):
        self.jsonify = data

    def __getattr__(self, attr):
        value = self.jsonify.get(attr)
        if isinstance(value, (list, dict)):
            return jsonify(value)
        return value

    def __getitem__(self, index):
        value = self.jsonify[index]
        if isinstance(value, (list, dict)):
            return jsonify(value)
        return value

    def __setitem__(self, index, value):
        self.jsonify[index] = value

    def __delattr__(self, index):
        self.jsonify.pop(index)

    def __delitem__(self, index):
        self.jsonify.pop(index)

    def __repr__(self):
        return json.dumps(self.jsonify, indent=2, default=lambda x: str(x))

exmaple

response = jsonify(
    {
        'test': {
            'test1': [{'ok': 1}]
        }
    }
)
response.test -> jsonify({'test1': [{'ok': 1}]})
response.test.test1 -> jsonify([{'ok': 1}])
response.test.test1[0] -> jsonify({'ok': 1})
response.test.test1[0].ok -> int(1)
Fiveandten answered 20/7, 2021 at 13:23 Comment(0)
S
0

Here is my way.

Features

  • support type hints
  • raise error if key is missing.
  • skip extra value in data
import typing

class User:
    name: str
    age: int

    def __init__(self, data: dict):
        for k, _ in typing.get_type_hints(self).items():
            setattr(self, k, data[k])

data = {
    "name": "Susan",
    "age": 18
}

user = User(data)
print(user.name, user.age)

# Output: Susan 18
Sturgis answered 22/2, 2022 at 3:31 Comment(0)
N
0

Using python 3.7, I find the following quite simple and effective. In this case, loading JSON from a file into a dictionary:

class Characteristic:
    def __init__(self, characteristicName, characteristicUUID):
        self.characteristicName = characteristicName
        self.characteristicUUID = characteristicUUID


class Service:
    def __init__(self, serviceName, serviceUUID, characteristics):
        self.serviceName = serviceName
        self.serviceUUID = serviceUUID
        self.characteristics = characteristics

class Definitions:
    def __init__(self, services):
        self.services = []
        for service in services:
            self.services.append(Service(**service))


def main():
    parser = argparse.ArgumentParser(
        prog="BLEStructureGenerator",
        description="Taking in a JSON input file which lists all of the services, "
                    "characteristics and encoded properties. The encoding takes in "
                    "another optional template services and/or characteristics "
                    "file where the JSON file contents are applied to the templates.",
        epilog="Copyright Brown & Watson International"
    )

    parser.add_argument('definitionfile',
                        type=argparse.FileType('r', encoding='UTF-8'),
                        help="JSON file which contains the list of characteristics and "
                             "services in the required format")
    parser.add_argument('-s', '--services',
                        type=argparse.FileType('r', encoding='UTF-8'),
                        help="Services template file to be used for each service in the "
                             "JSON file list")
    parser.add_argument('-c', '--characteristics',
                        type=argparse.FileType('r', encoding='UTF-8'),
                        help="Characteristics template file to be used for each service in the "
                             "JSON file list")

    args = parser.parse_args()
    definition_dict = json.load(args.definitionfile)
    definitions = Definitions(**definition_dict)
Nootka answered 24/1, 2023 at 3:58 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.