Nested Dictionaries to JSON for the POST request Python

Question

I am having trouble converting the payload data in the form of nested dictionaries to pass it as a data for the POST request using Python requests module. The form data is as below:

payload = {'request':  {
                'appkey': "936725A4-7D9A-11E5-81AC-86EC8D89CD5A"},
            'formdata':{
                    'currency':'US',
                    'dataview':'store_default',
                    'distinct':'_distance, clientkey',
                    'geolocs':{
                            'geoloc':[{
                                    '0':{
                                            'address1':'',
                                            'addressline':'19128, PA',
                                            'city':'Philadelphia',
                                            'country':'US',
                                            'latitude':'40.0532987',
                                            'longitude':'-75.23040379999998',
                                            'postalcode':'19128',
                                            'province':'',
                                            'state':'PA'}}]
                            },
                    'google_autocomplete':'true',
                    'limit':'250',
                    'nobf':'1',
                    'searchradius':'15|25|50|100|250|350|450|550|650|750|850|950',
                    'true':'1',
                    'where':{'partner_reseller': {'eq':'1'}}}                    
          }

r = requests.post(url,data=simplejson.dumps(payload),headers=header)
result = simplejson.loads(str(r.content))

Can somebody please assist me with structure and can point out the mistake in what I have written. I keep getting the following error:

{'code': 1008,
 'response': {'message': 'The submitted XML is not properly formed'}} 

I'll appreciate your help a lot. Thank you.

Thickhead

What API is this for? Is there documentation? The url you are hitting would help, if it is a public API – Sweatband

You got 'submitted XML' in message, so the api request probably XML data ? Or you should set in your header, the Content-Type to application/json for that. As said heinst, please give us the documentation or the name of the API if you can. – Trude

@rsz: I have set the 'content-type' to 'application/json' already in the header – Thickhead

Answer 1

I had similar problem, frustrating but I solved it. Python requests do not work with nested json, they expect one layer json. It processes like form(application/x-www-form-urlencoded)

# Wrong
data = {'param1': {'a':[100, 200]},
        'param2': 'value2',
        'param3': False}

# You have to convert values into string:
data = {'param1': json.dumps({'a':[100, 200]}),
        'param2': 'value2',
        'param3': json.dumps(False)}

In your case:

import json
params = {
        'appkey': "936725A4-7D9A-11E5-81AC-86EC8D89CD5A"},
        'formdata':{
            'currency':'US',
            'dataview':'store_default',
            'distinct':'_distance, clientkey',
            'geolocs':{
                    'geoloc':[{
                            '0':{
                                    'address1':'',
                                    'addressline':'19128, PA',
                                    'city':'Philadelphia',
                                    'country':'US',
                                    'latitude':'40.0532987',
                                    'longitude':'-75.23040379999998',
                                    'postalcode':'19128',
                                    'province':'',
                                    'state':'PA'}}]
                    },
            'google_autocomplete':'true',
            'limit':'250',
            'nobf':'1',
            'searchradius':'15|25|50|100|250|350|450|550|650|750|850|950',
            'true':'1',
            'where':{'partner_reseller': {'eq':'1'}}}                    
          }
payload = {'request':  json.dumps(params) }

r = requests.post(url,data=payload) # important to keep payload as json!!!
result = r.text # or depends what the return is..

Marabou

Thanks to this! Should be voted as accepted answer. – Crimea

Thank you, saved me from more than a week of frustration. I was not aware of the nested json limitation in requests. – Wilburn

Thank you so much. I spent so much time on this – Hardiness

Answer 2

My suggestion is to use the JSON parameter and let requests both encode the object to JSON and let requests set the Content-Type header to application/json.

It's very possible that the web service assumes you're passing it XML, unless you specify that you're passing JSON, via setting the Content-Type to application/json. (It's also possible this web API really wants XML too, the docs for the service would tell you)

requests.post(url,json=payload,headers=header)

Eudy

@RyanWicox: Thank you for your suggestion. So should I use the payload in the same form as I have mentioned here or do I need to change the structure? – Thickhead

I'm getting an following error: TypeError: request() got an unexpected keyword argument 'json'. – Thickhead

The json parameter was added in requests 2.4.2, are you using a version before that maybe???/ – Eudy

same error. I have the updated the package to version 2.13.0 – Thickhead

Answer 3

Taking inspiration from both the previous answers, another approach that I personally find cleaner is to serialize the body into string and set Content-Type header to application/json

import json, requests

#Other stuff

headers = { "Content-Type": "application/json" }
data = json.dumps(payload)

r = requests.post(url,data=data,headers=headers)

Anacreontic

Answer 4

You can use this before any post request.

import json, requests

if header.get('Content-Type') == 'application/x-www-form-urlencoded':
    params = {k: (json.dumps(v) if isinstance(v, dict) else v) for k, v in params.items()}

response = requests.post(url=url, headers=header, data=params)

Trader