Curl POST request into pycurl code

Question

I'm trying to convert following curl request into pycurl:

curl -v
-H Accept:application/json \
-H Content-Type:application/json \
-d "{
    name: 'abc',
    path: 'def',
    target: [ 'ghi' ]
}" \
-X POST http://some-url

I have following python code:

import pycurl, json

c = pycurl.Curl()
c.setopt(pycurl.URL, 'http://some-url')
c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
data = json.dumps({"name": "abc", "path": "def", "target": "ghi"})
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, data)
c.setopt(pycurl.VERBOSE, 1)
c.perform()
print curl_agent.getinfo(pycurl.RESPONSE_CODE)
c.close()

Executing this I had an error 415: Unsupported media type, so I have changed:

c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])

into:

c.setopt(pycurl.HTTPHEADER, [ 'Content-Type: application/json' , 'Accept: application/json'])

This time I have 400: Bad request. But bash code with curl works. Do you have any idea what should I fix in python code?

Goldthread

Answer 1

In your bash example, the property target is an array, in your Python example it is a string.

Try this:

data = json.dumps({"name": "abc", "path": "def", "target": ["ghi"]})

I also strongly advise you to check out the requests library which has a much nicer API:

import requests
data = {"name": "abc", "path": "def", "target": ["ghi"]}
response = requests.post('http://some-url', json=data)
print response.status_code

Farmhouse

The requests library is broken in many way as to be useless for any server that is strict or slightly non-conforming. There are requests that work perfectly fine with curl but that get rejected when made with the requests library due to its various unnecessary sanitizing. – Absenteeism

@Absenteeism thanks for your comment. I wasn't aware of that and never had any issues with requests myself, but would be interested in learning more about the problems you describe. Can you point me to any further information? – Farmhouse

No,i strongly disagree, python requests is not always a champion. If you want to upload a file in readable stream then pycurl's FORM_FILE comes to rescue. – Absinthe

Answer 2

PycURL is a wrapper on the libcurl library written in C language so its Python API can be bit puzzling. As some people are advocating use of python requests instead I just want to point out that it isn't a perfect replacement. For me, its lack of DNS resolution timeout was a deal breaker. I also find it much slower on my Raspberry Pi. This comparison may be relevant: Python Requests vs PyCurl Performance

So here's something that doesn't evade OP's question:

import pycurl
import json
from cStringIO import StringIO

curl = pycurl.Curl()
curl.setopt(pycurl.URL, 'http://some-url')
curl.setopt(pycurl.HTTPHEADER, ['Accept: application/json',
                                'Content-Type: application/json'])
curl.setopt(pycurl.POST, 1)

# If you want to set a total timeout, say, 3 seconds
curl.setopt(pycurl.TIMEOUT_MS, 3000)

## depending on whether you want to print details on stdout, uncomment either
# curl.setopt(pycurl.VERBOSE, 1) # to print entire request flow
## or
# curl.setopt(pycurl.WRITEFUNCTION, lambda x: None) # to keep stdout clean

# preparing body the way pycurl.READDATA wants it
# NOTE: you may reuse curl object setup at this point
#  if sending POST repeatedly to the url. It will reuse
#  the connection.
body_as_dict = {"name": "abc", "path": "def", "target": "ghi"}
body_as_json_string = json.dumps(body_as_dict) # dict to json
body_as_file_object = StringIO(body_as_json_string)

# prepare and send. See also: pycurl.READFUNCTION to pass function instead
curl.setopt(pycurl.READDATA, body_as_file_object) 
curl.setopt(pycurl.POSTFIELDSIZE, len(body_as_json_string))
curl.perform()

# you may want to check HTTP response code, e.g.
status_code = curl.getinfo(pycurl.RESPONSE_CODE)
if status_code != 200:
    print "Aww Snap :( Server returned HTTP status code {}".format(status_code)

# don't forget to release connection when finished
curl.close()

There are some more interesting features worth checking out in the libcurl curleasy setopts documentation

Streptomycin

The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively. – Version

Maybe useful:from io import StringIO – Version

@MaxBase Interesting but cStringIO has its own place as it is written in C unlike io.StringIO or StringIO.StringIO and it can be seen how well it performs in this benchmark A common use of PycURL which is written in C is to write higher performing code so I think readers would find cStringIO to be a better compliment. – Streptomycin

Thanks a lot, that helped me a lot after days of investigation – Joleen

Answer 3

I know this is over a year old now, but please try removing the whitespace in your header value.

c.setopt(pycurl.HTTPHEADER, ['Accept:application/json'])

I also prefer using the requests module as well because the APIs/methods are clean and easy to use.

Apostatize

Answer 4

I had similar problem, and I used your code example but updated the httpheader section as follows:

c.setopt(pycurl.HTTPHEADER, ['Content-Type:application/json'])

Abrasive

Answer 5

It's better simple to use requests library. (http://docs.python-requests.org/en/latest)

I append python code for your original curl custom headers.

import json
import requests

url = 'http://some-url'
headers = {'Content-Type': "application/json; charset=xxxe", 'Accept': "application/json"}
data = {"name": "abc", "path": "def", "target":  ["ghi"]}
res = requests.post(url, json=data, headers=headers)
print (res.status_code)
print (res.raise_for_status())

Trichromat

Please explain why it's better – Stalagmite