How to merge two json string in Python?

Question

I recently started working with Python and I am trying to concatenate one of my JSON String with existing JSON String. I am also working with Zookeeper so I get the existing json string from zookeeper node as I am using Python kazoo library.

# gets the data from zookeeper
data, stat = zk.get(some_znode_path)
jsonStringA = data.decode("utf-8")

if I print jsonStringA it gives me like this -

{"error_1395946244342":"valueA","error_1395952003":"valueB"}

But if I do print json.loads(jsonString) then it prints out like this -

{u'error_1395946244342': u'valueA', u'error_1395952003': u'valueB'}

Here jsonStringA will have my existing JSON String. Now I have another key-value pair which I need to add in the exiting jsonStringA -

Below is my Python code -

# gets the data from zookeeper
data, stat = zk.get(some_znode_path)
jsonStringA = data.decode("utf-8")

timestamp_in_ms = "error_"+str(int(round(time.time() * 1000)))
node = "/pp/tf/test/v1"
a,b,c,d = node.split("/")[1:]
host_info = "h1"
local_dc = "dc3"
step = "step2"

My existing jsonStringA will be like this after extracting from zookeeper -

{"error_1395946244342":"valueA","error_1395952003":"valueB"}

Now I need to append this key-value pair in the jsonStringA -

"timestamp_in_ms":"Error Occured on machine "+host_info+" in datacenter "+ local_dc +" on the "+ step +" of process "+ c +"

So in short I need to merge below key-value pair -

"error_1395952167":"Error Occured on machine h1 in datacenter dc3 on the step2 of process test"

So final JSON String will look like this -

{"error_1395946244342":"valueA","error_1395952003":"valueB","error_1395952167":"Error Occured on machine h1 in datacenter dc3 on the step2 of process test"}

Is this possible to do?

Landaulet

Answer 1

Assuming a and b are the dictionaries you want to merge:

c = {key: value for (key, value) in (a.items() + b.items())}

To convert your string to python dictionary you use the following:

import json
my_dict = json.loads(json_str)

---

Update: full code using strings:

# test cases for jsonStringA and jsonStringB according to your data input
jsonStringA = '{"error_1395946244342":"valueA","error_1395952003":"valueB"}'
jsonStringB = '{"error_%d":"Error Occured on machine %s in datacenter %s on the %s of process %s"}' % (timestamp_number, host_info, local_dc, step, c)

# now we have two json STRINGS
import json
dictA = json.loads(jsonStringA)
dictB = json.loads(jsonStringB)

merged_dict = {key: value for (key, value) in (dictA.items() + dictB.items())}

# string dump of the merged dict
jsonString_merged = json.dumps(merged_dict)

But I have to say that in general what you are trying to do is not the best practice. Please read a bit on python dictionaries.

---

Alternative solution:

jsonStringA = get_my_value_as_string_from_somewhere()
errors_dict = json.loads(jsonStringA)

new_error_str = "Error Ocurred in datacenter %s blah for step %s blah" % (datacenter, step)
new_error_key = "error_%d" % (timestamp_number)

errors_dict[new_error_key] = new_error_str

# and if I want to export it somewhere I use the following
write_my_dict_to_a_file_as_string(json.dumps(errors_dict))

And actually you can avoid all these if you just use an array to hold all your errors.

Liebman

do you work with strings or dictionaries? The example I gave you is correct. – Liebman

I updated the question again. As I recently started with Python so not sure what does dictionary means here. I have updated the question with how my jsonStringA gets printed out as well. The string which I need to append is a simple string I guess. – Landaulet

i updated my answer. now I use plain json strings as input. please read it carefully so you can understand what is happening with the strings and dictionaries. – Liebman

That makes sense now.. One last question how do I make jsonStringB with all its values properly? As I need to make a key value pair with all the values from the variable. – Landaulet

see updated code. %s adds in place a parameter of type string, %d adds an integer. All the parameters are in the tuple at the end. (a, b, c,..) – Liebman

I hope it helps, please mark the question as answered if you are satisfied of the outcome. – Liebman

Sure I will. But can you please tell me what's wrong with this approach and why you are saying this is not the right way? Just curious to know. – Landaulet

The keys seem that they don't have big importance other than the priority provided by the timestamp. So it's more complicated to maintain dictionaries than arrays. Especially if you have all of this string conversions/json decoding. Except if you need this data structure at later stage not shown in the above code sample. Personally, in terms of coding, I like the idea of "Keep It Simple Stupid", and of course, my favourite, "Less is More" :) – Liebman

Thanks Vame for the help. Yes they have some use. I will write this json string back to Zookeeper and then I have a Java program running which is keeping a watch on that node and extract only what is latest timestamp. Anyways, tell me one thing suppose if this method get_my_value_as_string_from_somewhere() returns empty string then will I be able to make the proper json string using new_error_str? – Landaulet

No you cannot decode an empty string into json. You can do something like jsonStringA = get_my_value_as_string_from_somewhere() or '{}'. {} is a valid json object. – Liebman

make sense.. But in this case, how do I check if jsonStringA is empty, then assign jsonStringA to {}? Otherwise keep the value as it is. Is this possible to do? – Landaulet

what you can see in my last comment is what you want to do. they python way :) – Liebman

This answer is outdated and will not work (as is) for Python 3. In Py 2, dict.items() returns a list, but in Py 3, dict.items() returns a dict_items. Simplest (but inefficient) workaround is to cast to list: list(dict.items()). See also @Pi's answer. – Dermatome

Also, note that this will silently handle key collisions by overwriting values in a with values in b. – Dermatome

Answer 2

As of Python 3.5, you can merge two dicts with:

merged = {**dictA, **dictB}

(https://www.python.org/dev/peps/pep-0448/)

So:

jsonMerged = {**json.loads(jsonStringA), **json.loads(jsonStringB)}
asString = json.dumps(jsonMerged)

etc.

Acetum

this does not work if there are nested dictionaries – Laurent

do note that dictionary unpacking will eat up large resource – Wallow

Please note that this will silently handle key collisions by overwriting values in dictA with values in dictB. – Dermatome

Answer 3

You can load both json strings into Python Dictionaries and then combine. This will only work if there are unique keys in each json string.

import json

a = json.loads(jsonStringA)
b = json.loads(jsonStringB)
c = dict(a.items() + b.items())
# or c =  dict(a, **b)

Careless

Please don't use dict(a, **b). It's actually an abuse of the CPython implementation. Rather use c = a.copy(); c.update(b). – Lauralauraceous

Tried this in python3 and got TypeError: unsupported operand type(s) for +: 'dict_items' and 'dict_items'. – Comedian

Answer 4

Merging json objects is fairly straight forward but has a few edge cases when dealing with key collisions. The biggest issues have to do with one object having a value of a simple type and the other having a complex type (Array or Object). You have to decide how you want to implement that. Our choice when we implemented this for json passed to chef-solo was to merge Objects and use the first source Object's value in all other cases.

This was our solution:

from collections import Mapping
import json


original = json.loads(jsonStringA)
addition = json.loads(jsonStringB)

for key, value in addition.iteritems():
    if key in original:
        original_value = original[key]
        if isinstance(value, Mapping) and isinstance(original_value, Mapping):
            merge_dicts(original_value, value)
        elif not (isinstance(value, Mapping) or 
                  isinstance(original_value, Mapping)):
            original[key] = value
        else:
            raise ValueError('Attempting to merge {} with value {}'.format(
                key, original_value))
    else:
        original[key] = value

You could add another case after the first case to check for lists if you want to merge those as well, or for specific cases when special keys are encountered.

Wein

In my case, all the json string has key:value pair and they all are strings. I don't have any arrays stuff.. – Landaulet

Then don't worry about it, this code will work fine as is in that case. – Wein

One last question, how do I make json_string2 as the key value pair in my case? As I need to construct this json string from the variable I have. Any thoughts? – Landaulet

Ah, I think I misread your code, I believe your jsonString[AB] are actually my code's json_string[12]. I'll edit my code to reflect that assumption. – Wein

Answer 5

To append key-value pairs to a json string, you can use dict.update: dictA.update(dictB).

For your case, this will look like this:

dictA = json.loads(jsonStringA)
dictB = json.loads('{"error_1395952167":"Error Occured on machine h1 in datacenter dc3 on the step2 of process test"}')

dictA.update(dictB)
jsonStringA = json.dumps(dictA)

Note that key collisions will cause values in dictB overriding dictA.

Dermatome

Answer 6

What do you mean by merging? JSON objects are key-value data structure. What would be a key and a value in this case? I think you need to create new directory and populate it with old data:

d = {}
d["new_key"] = jsonStringA[<key_that_you_did_not_mention_here>] + \ 
               jsonStringB["timestamp_in_ms"]

Merging method is obviously up to you.

Butts