Exclude empty/null values from JSON serialization

Question

I am serializing multiple nested dictionaries to JSON using Python with simplejson.

Is there any way to automatically exclude empty/null values?

For example, serialize this:

 {
     "dict1" : {
     "key1" : "value1",
     "key2" : None
     }
 }

to

 {
     "dict1" : {
     "key1" : "value1"
     }
 }

When using Jackson with Java you can use Inclusion.NON_NULL to do this. Is there a simplejson equivalent?

Del

Answer 1

def del_none(d):
    """
    Delete keys with the value ``None`` in a dictionary, recursively.

    This alters the input so you may wish to ``copy`` the dict first.
    """
    # For Python 3, write `list(d.items())`; `d.items()` won’t work
    # For Python 2, write `d.items()`; `d.iteritems()` won’t work
    for key, value in list(d.items()):
        if value is None:
            del d[key]
        elif isinstance(value, dict):
            del_none(value)
    return d  # For convenience

Sample usage:

>>> mydict = {'dict1': {'key1': 'value1', 'key2': None}}
>>> print(del_none(mydict.copy()))
{'dict1': {'key1': 'value1'}}

Then you can feed that to json.

Griffey

Em.... That produces RuntimeError: dictionary changed size during iteration with Python 3.5 – Margetmargette

def del_none(d): """ Delete keys with the value ``None`` and empty string in a dictionary, recursively. This does not alter input, but copies the dictionary. Child dictionaries are also copied. No other objects are copied. """ rez = d.copy() for key, value in d.items(): if value is None or value == '': del rez[key] elif isinstance(value, dict): rez[key] = del_none(value) return rez – Margetmargette

@AleksandrPanzin: I’ve updated the code to target Python 3 with notes for Python 2. This was written seven years ago! I left it as an in-place modification, however. – Griffey

Watch out... none of these answers recurse into lists. Dictionaries appearing in lists will continue to be serialized with any null values present. – Kruller

This still isn't part of the python standard library? Color me surprised! – Carpio

I tried list(d.items()) with latest versions of 2.7 (and all the 3.x versions), and seems to work fine. So they might have fixed this in a later 2.7 version – Virelay

On Python 2, dict.items produces a list, dict.iteritems an iterator. So list(d.items()) always worked, but it’s needlessly inefficient: d.items() produces a list, then list(_) duplicates it unnecessarily. d.items() or list(d.iteritems()) will skip that needless duplication. Since we’re here, I’ll also remind you and others that Python 2 is EOL; no maintained code should depend on any version older than 3.6 at the time of writing. – Griffey

Answer 2

My Python3 version of this has the benefit of not changing the input, as well as recursion into dictionaries nested in lists:

def clean_nones(value):
    """
    Recursively remove all None values from dictionaries and lists, and returns
    the result as a new dictionary or list.
    """
    if isinstance(value, list):
        return [clean_nones(x) for x in value if x is not None]
    elif isinstance(value, dict):
        return {
            key: clean_nones(val)
            for key, val in value.items()
            if val is not None
        }
    else:
        return value

For example:

a = {
    "a": None,
    "b": "notNone",
    "c": ["hello", None, "goodbye"],
    "d": [
        {
            "a": "notNone",
            "b": None,
            "c": ["hello", None, "goodbye"],
        },
        {
            "a": "notNone",
            "b": None,
            "c": ["hello", None, "goodbye"],
        }
    ]
}


print(clean_nones(a))

results in this:

{
    'b': 'notNone',
    'c': ['hello', 'goodbye'],
    'd': [
        {
            'a': 'notNone',
            'c': ['hello', 'goodbye']
        },
        {
            'a': 'notNone',
            'c': ['hello', 'goodbye']
        }
    ]
}

Cablegram

This should be the accepted answer as the question was about json and json allows objects in lists! – Armand

Answer 3

>>> def cleandict(d):
...     if not isinstance(d, dict):
...         return d
...     return dict((k,cleandict(v)) for k,v in d.iteritems() if v is not None)
... 
>>> mydict = dict(dict1=dict(key1='value1', key2=None))
>>> print cleandict(mydict)
{'dict1': {'key1': 'value1'}}
>>> 

I don't like using del in general, changing the existing dictionary can have subtle effects depending on how they are created. Creating new dictionaries with None removed prevents all side effect.

Daradarach

d.items() in Python 3.x – Superfluous

changing to d.items() works for Python3, so long as your structure doesn't contain a list – Donough

Answer 4

You can try this approach. In my case (I use python 3), it works well.

def to_json(self):
    return json.dumps(self,
                      default=lambda o: dict((key, value) for key, value in o.__dict__.items() if value),
                      indent=4,
                      allow_nan=False)

Aflutter

Answer 5

This solution is correction of the one above from @eric which does not handle list type corectly.

Values in canonical JSON dictionary can be of one of following 3 types:

• dictionary
• list
• value type (string, integer or floating point)

Note: Assumption is that we are dealing here with canonical JSON dictionary which can really contain only above mentioned types. If dictionary contains other types then ones mentioned above (e.g. tuples, custom classes, ...), then this solution won't work as expected.

The essential difference between this solution (below) and the original one from @eric is that list can contain elements of dictionary type from iside of which we want to drop elements with None value.

def cleandict(d):
    if isinstance(d, dict):
        return {k: cleandict(v) for k, v in d.items() if v is not None}
    elif isinstance(d, list):
        return [cleandict(v) for v in d]
    else:
        return d

Note: Please keep in mind that we must NOT remove None elements from the list since it would affect structural integrity of the list data. If some ( or all) of list elements have None value, they shall remain listed in the list structure as they were in order to preserve original structural meaning/integrity of the list.

Hammerless

Answer 6

def excludeNone(d):
    for k in list(d):
        if k in d:
            if type(d[k]) == dict:
                excludeNone(d[k])
            if not d[k]:
                del d[k]

Chorale

It would be safer to use if d[k] is not None instead of if not d[k] – otherwise empty strings and 0 values will also be excluded from the output. – Tsui

Answer 7

It works for me:

When dictionary has dict/list/tuple values ....

for example it is my object:

dict_obj = {
    'inline_keyboard': [
        [
            {'text': '0-0', 'url': None, 'login_url': None, 'callback_data': '0-0', 'switch_inline_query': None},
            {'text': '0-1', 'url': None, 'login_url': None, 'callback_data': '0-1', 'switch_inline_query': None}
        ],
        [
            {'text': '1-0', 'url': None, 'login_url': None, 'callback_data': '1-0', 'switch_inline_query': None},
            {'text': '1-1', 'url': None, 'login_url': None, 'callback_data': '1-1', 'switch_inline_query': None}
        ],
        [
            {'text': '2-0', 'url': None, 'login_url': None, 'callback_data': '2-0', 'switch_inline_query': None}
        ]
    ]
}

I wrote this function:

def delete_none_values(obj):
    if isinstance(obj, dict):
        for k, v in list(obj.items()):
            if v is None:
                del obj[k]
            elif isinstance(v, dict):
                delete_none_values(v)
            elif isinstance(v, (list, tuple)):
                for _ in v:
                    delete_none_values(_)
    elif isinstance(obj, (list, tuple)):
        for _ in obj:
            delete_none_values(_)
    return obj

And then when use this fuction:

from json import dumps

print(
    dumps(
        delete_none_values(dict_obj.copy()),
        indent=2
    )
)

output is:

{
  "inline_keyboard": [
    [
      {"text": "0-0", "callback_data": "0-0"},
      {"text": "0-1", "callback_data": "0-1"}
    ],
    [
      {"text": "1-0", "callback_data": "1-0"},
      {"text": "1-1", "callback_data": "1-1"}
    ],
    [
      {"text": "2-0", "callback_data": "2-0"}
    ]
  ]
}

Walkin

Answer 8

Could you maybe remain 'url' if it has value in one place and remove it if it none on another place?

'inline_keyboard': [
        [
            {'text': '0-0', 'url': 'someValue', 'login_url': None, 'callback_data': '0-0', 'switch_inline_query': None},
            {'text': '0-1', 'url': None, 'login_url': None, 'callback_data': '0-1', 'switch_inline_query': None}
        ],
        [
            {'text': '1-0', 'url': None, 'login_url': None, 'callback_data': '1-0', 'switch_inline_query': None},
            {'text': '1-1', 'url': None, 'login_url': None, 'callback_data': '1-1', 'switch_inline_query': None}
        ],
        [
            {'text': '2-0', 'url': None, 'login_url': None, 'callback_data': '2-0', 'switch_inline_query': None}
        ]
]

Dispensatory

Answer 9

I used the MatanRubin function and extended it so that it also filters Nan (float) and Null (string) to be able to talk with a Php API.

from math import isnan

def clean_nones(value):
    """
    Recursively remove all None values from dictionaries and lists, and returns
    the result as a new dictionary or list.
    """
    def checkNan(value):
        if isinstance(value, float) and isnan(value):
            return True if (isinstance(value, float) and isnan(value)) else False
        
    if isinstance(value, list):
        return [clean_nones(x) for x in value if (x is not None and  x != 'NULL' and not checkNan(x))]
    elif isinstance(value, dict):
        return {
            key: clean_nones(val)
            for key, val in value.items()
            if (val is not None and val != 'NULL' and not checkNan(val))
        }
    else:
        return value

cleanedJson = clean_nones(toCleanJson)

Vehicle