L

34

1398

I have a basic dict as follows:

sample = {}
sample['title'] = "String"
sample['somedate'] = somedatetimehere

When I try to do jsonify(sample) I get:

TypeError: datetime.datetime(2012, 8, 8, 21, 46, 24, 862000) is not JSON serializable

What can I do such that my dictionary sample can overcome the error above?

Note: Though it may not be relevant, the dictionaries are generated from the retrieval of records out of mongodb where when I print out str(sample['somedate']), the output is 2012-08-08 21:46:24.862000.

Lingam answered 9/8, 2012 at 2:2 Comment(10)

Is this specifically python in general, or possibly django? – Insuperable 9/8, 2012 at 2:5

It technically is specifically python, I am not using django, but retrieving records out of mongodb. – Lingam 9/8, 2012 at 2:5

possible duplicate of JSON datetime between Python and JavaScript – Insuperable 9/8, 2012 at 2:5

I am using mongoengine, but if pymongo has better ways of getting around this or overcoming this, please tell. – Lingam 9/8, 2012 at 2:7

@Insuperable That answer you linked to as duplicate appears to makes no sense to me. If you can post an answer here that would make sense in my context, that would be more helpful. – Lingam 9/8, 2012 at 2:8

That duplicate I posted is the generic way to handle this where you specify the extra parameter to JSON as a handler. I have posted a pymongo specific answer. – Insuperable 9/8, 2012 at 2:10

The linked question is essentially telling you not to try to serialize the datetime object, but rather to convert it to a string in the common ISO format before serializing. – Nigercongo 9/8, 2012 at 2:13

For a generic solution that handles more that just datetime.datetime, see Making object JSON serializable with regular encoder. – Alacrity 19/6, 2017 at 12:57

cannot import name json_util – Alienee 1/2, 2018 at 5:9

If you have the option to store the datetime as a float (seconds since epoch, as from time.time()) that is portable and you can call datetime.datetime.fromtimestamp when you need a python datetime. – Selfdetermination 17/3, 2018 at 21:48

I

555

Updated for 2018

The original answer accommodated the way MongoDB "date" fields were represented as:

{"$date": 1506816000000}

If you want a generic Python solution for serializing datetime to json, check out @jjmontes' answer for a quick solution which requires no dependencies.

As you are using mongoengine (per comments) and pymongo is a dependency, pymongo has built-in utilities to help with json serialization:
http://api.mongodb.org/python/1.10.1/api/bson/json_util.html

Example usage (serialization):

from bson import json_util
import json

json.dumps(anObject, default=json_util.default)

Example usage (deserialization):

json.loads(aJsonString, object_hook=json_util.object_hook)

Django

Django provides a native DjangoJSONEncoder serializer that deals with this kind of properly.

See https://docs.djangoproject.com/en/dev/topics/serialization/#djangojsonencoder

from django.core.serializers.json import DjangoJSONEncoder

return json.dumps(
  item,
  sort_keys=True,
  indent=1,
  cls=DjangoJSONEncoder
)

One difference I've noticed between DjangoJSONEncoder and using a custom default like this:

import datetime
import json

def default(o):
    if isinstance(o, (datetime.date, datetime.datetime)):
        return o.isoformat()

return json.dumps(
  item,
  sort_keys=True,
  indent=1,
  default=default
)

Is that Django strips a bit of the data:

 "last_login": "2018-08-03T10:51:42.990", # DjangoJSONEncoder 
 "last_login": "2018-08-03T10:51:42.990239", # default

So, you may need to be careful about that in some cases.

Insuperable answered 9/8, 2012 at 2:7 Comment(20)

Is it good/bad practice to be mixing multiple libraries i.e. having mongoengine for inserting docs and pymongo for query/retrieval? – Lingam 9/8, 2012 at 2:25

Its not bad practice, it just implies some dependency on the libraries that your main library uses. If you can't accomplish what you need from mongoengine, then you drop down to pymongo. Its the same with Django MongoDB. With the later, you would try to stay within the django ORM to maintain backend agnostic state. But sometimes you can't do what you need in the abstraction, so you drop down a layer. In this case, its completely unrelated to your problem since you are just using utility methods to accompany the JSON format. – Insuperable 9/8, 2012 at 2:29

I am trying this out with Flask and it appears that by using json.dump, I am unable to put a jsonify() wrapper around it such that it returns in application/json. Attempting to do return jsonify(json.dumps(sample, default=json_util.default)) – Lingam 9/8, 2012 at 2:39

I am assuming jsonify would try and serialize it again. Is it a flask function specifically? – Insuperable 9/8, 2012 at 2:41

I think it is a flask function specifically to get it to return of "content-type: application/json" that usually accepts a dict, but I get: "ValueError: dictionary update sequence element #0 has length 1; 2 is required". I hope django is not better for this sort of thing... – Lingam 9/8, 2012 at 2:43

You would just need to use the flask general response object and set the content-type – Insuperable 9/8, 2012 at 2:43

@amit Its not so much about memorizing syntax, as it is about getting good at reading documentation and storing enough info in my head to recognise where and when I need to retrieve it again. In this case, one might say "Oh a custom object with json" and then quickly refresh on that usage – Insuperable 2/7, 2015 at 19:12

I had been using this method bson.json_util.dumps(anObject), but your example method json.dumps(anObject, default=bson.json_util.default) is exponentially faster. Thank you! – Yashmak 24/3, 2017 at 19:59

This will convert datetime to a dict with $date, not I wanted: json.dumps({'d':d}, default=json_util.default) -> '{"d": {"$date": 1506816000000}}' – Biparous 7/11, 2017 at 10:13

@Biparous I haven't tracked changes in bjson or mongo since I wrote this 5 years ago. But if you want control over the serialisation of the datetime then you need to write your own default handler function as illustrated in the answer given by jgbarah – Insuperable 7/11, 2017 at 10:24

The answer by jgbarah doesn't have any example for the object_hook, and the bjson source code doesn't show any sign of json_util or object_hook, 5 years ago or today. Can anyone point me to an example? I fundamentally don't know input/return structure is expected except for just guessing and checking. – Dishearten 8/3, 2018 at 15:55

@Dishearten the bson module is being provided by pymongo, so you can see the source here: github.com/mongodb/mongo-python-driver/blob/master/bson/… – Insuperable 8/3, 2018 at 18:22

in the personal don't like use external librarys, i prefer the utisl from python as example datetime, isoformat()..etc.. – Gigahertz 23/4, 2018 at 18:55

@DiegoAvila sure under normal circumstances it is wise to always evaluate your dependencies. But in this case the question was about the Python mongodb client which comes with the needed serialisation facilities. – Insuperable 23/4, 2018 at 19:19

Also in Django, if you have this problem with model serialization in a JSONField (from django.contrib.postgres.fields), the solution is to add encoder=DjangoJSONEncoder. ie: my_field = JSONField(... encoder=DjangoJSONEncoder) – Nedanedda 27/2, 2019 at 4:37

The question is not tagged as either Mongo or Django...but low and behold. – Chemosynthesis 26/6, 2019 at 6:2

@Chemosynthesis yea thought the immediate questions that followed in the comments clarified that it was a mongodb and Django situation – Insuperable 27/6, 2019 at 8:49

When I am using json.loads(aJsonString, object_hook=json_util.object_hook) I am getting Object of type ObjectId is not JSON serializable – Steapsin 8/2, 2021 at 11:42

@NilaySingh that sounds similar to this other question that I answered. Might want to check your json string format and make sure it uses "$oid" and not "ObjectId()" in the string form: https://mcmap.net/q/41801/-unable-to-deserialize-pymongo-objectid-from-json – Insuperable 9/2, 2021 at 18:29

@Insuperable I have a complex json object with a timedelta format and it is only working with DjangoJSONEncoder I am not sure if I can any other default ways or not. What do you think? – Wellman 19/10, 2021 at 12:31

N

1717

My quick & dirty JSON dump that eats dates and everything:

json.dumps(my_dictionary, indent=4, sort_keys=True, default=str)

default is a function applied to objects that aren't serializable.
In this case it's str, so it just converts everything it doesn't know to strings. Which is great for serialization but not so great when deserializing (hence the "quick & dirty") as anything might have been string-ified without warning, e.g. a function or numpy array.

Nigrosine answered 21/3, 2016 at 22:29 Comment(17)

This is awesome, But unfortunately I did not understand what happened? Can anyone explain this answer? – Keratinize 4/10, 2016 at 5:31

@KishorPawar: default is a function applied to objects that aren't serializable. In this case it's str, so it just converts everything it doesn't know to strings. Which is great for serialization but not so great when deserializing (hence the "quick & dirty") as anything might have been string-ified without warning, e.g. a function or numpy array. – Phasis 19/10, 2016 at 20:54

@Phasis awesome. Thanks. Useful when you know type of those non-serializable values like dates. – Keratinize 20/10, 2016 at 0:32

Can someone share how to return back to original dictionary? – Dubitable 5/12, 2018 at 7:34

repr might have a slight advantage over str, but of course there are no guarantees. – Unruh 21/3, 2019 at 23:12

@jjmontes, doesn't work for everything, eg json.dumps({():1,type(None):2},default=str) raises TypeError, can't have type or tuple. – Outrank 22/8, 2019 at 18:46

@c-urchin, repr has same issues as str for examples in above comment. – Outrank 22/8, 2019 at 18:47

I am sorry after this comment only i found that its syntax mistake and not of default=str. Thank you for this – Anuska 29/8, 2019 at 22:24

Had to wrap mine in print(...) to get it to pretty print. – Mathews 27/1, 2020 at 5:35

This saved a year of my life. I would be cautious to use the indent when sending data over APIs – Supernova 13/5, 2020 at 5:20

This works :-), but when testing to use this also for serialisable data and writing the resulting string to file using json.dump(), I find that my whole data set is enclosed in double quotes, and the double quotes around the separate elements are (then logically) escaped. Is there a way to prevent that? – Valaria 24/8, 2020 at 15:35

@DannyBullis json has pprint built-in: json.dumps(x, default=str, indent=4) – Ringer 19/10, 2020 at 16:59

@Phasis "but not so great when deserializing" - it returns a standartized parseable string. Do you know better human-readable json compatible representation? – Sciurine 12/1, 2021 at 3:42

@SmitJohnth I'm glad you ask, because this is why I made json-tricks: github.com/mverleg/pyjson_tricks – Phasis 15/1, 2021 at 21:49

man, this should be the default in Python – Woodpile 20/1, 2021 at 9:56

Django users: docs.djangoproject.com/en/3.2/topics/serialization/… – Argentum 15/11, 2021 at 12:2

I have to wonder why default does not default to str in the first place. – Nanny 19/12, 2022 at 18:35

D

627

Building on other answers, a simple solution based on a specific serializer that just converts datetime.datetime and datetime.date objects to strings.

from datetime import date, datetime

def json_serial(obj):
    """JSON serializer for objects not serializable by default json code"""

    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    raise TypeError ("Type %s not serializable" % type(obj))

As seen, the code just checks to find out if object is of class datetime.datetime or datetime.date, and then uses .isoformat() to produce a serialized version of it, according to ISO 8601 format, YYYY-MM-DDTHH:MM:SS (which is easily decoded by JavaScript). If more complex serialized representations are sought, other code could be used instead of str() (see other answers to this question for examples). The code ends by raising an exception, to deal with the case it is called with a non-serializable type.

This json_serial function can be used as follows:

from datetime import datetime
from json import dumps

print dumps(datetime.now(), default=json_serial)

The details about how the default parameter to json.dumps works can be found in Section Basic Usage of the json module documentation.

Dabney answered 6/3, 2014 at 23:46 Comment(11)

yeah the correct answer, more pretty import datetime and if isinstance(obj, datetime.datetime) , I lost many time because don't used from datetime import datetime , anyway thanks – Leiker 22/9, 2014 at 20:43

but this does not explain how to deserialise it with the correct type, isnt it ? – Oconnor 26/6, 2015 at 15:53

No, @Oconnor , nothing said about that. In my case, I'm deserializing in JavaScript, which works out of the box. – Dabney 28/6, 2015 at 8:9

This will cause unexpected behavior if the json module ever updates to include serialization of datetime objects. – Jamisonjammal 16/8, 2016 at 17:27

I'm worried that the error messages this replaces will be less helpful, and that it woudln't work on non-datatime values at all. I don't fully understand the api this working with. All I expect is that you'd have to call the base class in the default case like https://mcmap.net/q/41626/-how-can-i-overcome-quot-datetime-datetime-not-json-serializable-quot does. – Gorky 27/9, 2016 at 22:43

Keep in mind that obj.isoformat() (and str(obj)) will output the timezone info (like -08:00), so if the same date was defined in different time zones their json representations won't match. The solution would be to convert dates to either UTC or timestamps. – Egin 6/9, 2017 at 18:0

"If specified, default should be a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError. If not specified, TypeError is raised." Thanks. – Gayden 2/4, 2018 at 9:53

@Egin But converting times to UTC would unify 01:00:00+01:00 and 02:00:00+00:00 which are not supposed to be the same, depending on context. They refer to the same point in time of course, but the offset might be a relevant aspect of the value. – Doucet 18/9, 2019 at 12:26

+1 for supporting datetime.date. However, maybe even better idea would be to use duck typing instead of isinstance, to support all obj that implements isoformat() method (e.g. pendulum and arrow)? – Idiot 19/5, 2020 at 10:11

How would you do that in a nested dictionary that somewhere has datetime? – Manganin 30/8, 2022 at 13:21

Do you lose any information from the datetime that doesn't get into the isoformat string? – Radiobroadcast 1/12, 2023 at 17:15

I

555