Python httplib and POST

Question

I am currently working with a piece of code that has been written by somebody else. It uses httplib to make requests to server. It has all the data supplied in a correct format - for example message body, header values, etc.

The problem is that each time it attempts to send a POST requests, the data is there - I can see it on the client side, however nothing arrives to the server. I've read through the library specification and the usage seems to be correct.

The extracted library calls go as follows:

import httplib

conn = httplib.HTTPConnection('monkeylabs.pl', 80)
conn.connect()
request = conn.putrequest('POST', '/api/snippet/')
headers = {}
headers['Content-Type'] = 'application/json'
headers['User-Agent'] = 'Envjs/1.618 (SpyderMonkey; U; Linux x86_64 2.6.38-10-generic;  pl_PL.utf8; rv:2.7.1) Resig/20070309 PilotFish/1.3.pre03'
headers['Accept'] = '*/*'
for k in headers:
    conn.putheader(k, headers[k])
conn.endheaders()

conn.send('[{"id":"route"}]')

resp = conn.getresponse()
print resp.status
print resp.reason
print resp.read()

conn.close()

Is this some known issue, or what? I'm using Python 2.7. Not sure how to check the version of httplib.

Please don't suggest to exchange httplib for something else unless it's something really similar (httplib2 perhaps). As I said, the code isn't mine and it comes in much greater amounts than what I've just posted above. Refactoring it would cause a major problem. I'm interested in any reliable workarounds.

EDIT

The debug output:

send: 'POST /api/snippet/ HTTP/1.1\r\nHost: monkeylabs.pl\r\nAccept-Encoding: identity\r\nContent-Type: application/json\r\nAccept: */*\r\nUser-Agent: Envjs/1.618 (SpyderMonkey; U; Linux x86_64 2.6.38-10-generic; pl_PL.utf8; rv:2.7.1) Resig/20070309 PilotFish/1.3.pre03\r\n\r\n[{"id":"route"}]'
reply: 'HTTP/1.0 201 CREATED\r\n'
header: Date: Fri, 10 Jun 2011 23:54:00 GMT
header: Server: WSGIServer/0.1 Python/2.7.1+
header: Vary: Cookie
header: Content-Type: application/json
header: Content-Length: 0
201
CREATED

Note that the information after reply actually talks about the server reply, not the request itself, which in this case is empty. The primary cause is that the request body itself is empty which I can observe by getting a log:

[11/Jun/2011 01:54:00] "POST /api/snippet/ HTTP/1.1" 201 0

And those three lines:

``
<QueryDict: {}>
<QueryDict: {}>

out of:

print '`%s`' % request.raw_post_data
print request.GET
print request.POST

on the Django server. So it seems it attempts to send the body but doesn't send it in the end.

EDIT(2)

Ok, I took a dump and it indeed told me that in the message sent from the browser there is an additional parameter called 'Content-Length' which has been omitted in the regular usage of the library. Silly me.

Wirth

BTW, if you think you'll be debugging this thing a lot, install Wireshark and use it to spy on what's actually going over the wire. – Furiya

Thanks, I was a non-believer at the beginning but it actually led me to the solution. Thanks again – Wirth

The server has: response.set_header('Access-Control-Allow-Origin', '*') ???? – Mercola

Answer 1

try adding:

conn.set_debuglevel(1)

to your code so you can see what is actually happening.

Uppercut

Answer 2

The putrequest method does not automatically add the Content Length header, you need to do it yourself or use the request method.

Add this to your code above the for loop:

headers['Content-Length'] = "%d"%(len('[{"id":"route"}]'))

Thenceforth

Answer 3

Have you checked the httplib documentation? httplib is a standard Python library, and Python is also very good about having online documentation: http://docs.python.org/library/httplib.html

The example from that page:

>>> import httplib, urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> headers = {"Content-type": "application/x-www-form-urlencoded",
...            "Accept": "text/plain"}
>>> conn = httplib.HTTPConnection("musi-cal.mojam.com:80")
>>> conn.request("POST", "/cgi-bin/query", params, headers)
>>> response = conn.getresponse()
>>> print response.status, response.reason
200 OK
>>> data = response.read()
>>> conn.close()

Your example appears to be more work than this, and if you read the API documentation -- for instance for putrequest -- you'll see you're using it wrong in your sample. Specifically, it automatically adds the Accept header by default.

Try making your code more like the functional example, also try to understand the calls used in both your code and used in the working example.

Keare

Yes, I did read the API documentation and everything seems fine. The problem is that it isn't fine. The same request made by various other means - say a browser or another library works well. Removing 'Accept' header doesn't change anything. The example that you copy pasted from the API is just doing something totally different. It's not a POST and it doesn't attempt to upload the message body. – Wirth