How do I access the data sent to my server using BaseHTTPRequestHandler? [duplicate]

Question

I'm a newbie to Python (using v3.3) and web programing and I've been struggling with a problem all night. I'm issuing a POST call to my server and sending it some data as follows:

DATA = {"listName":"Test list","listDesc":"A test list with test stuff in it.","refreshMode":"Replace","DBKey":"1","UserDisplaySeq":"1"}
DATA = json.dumps(DATA)
METHOD = "POST"
DATA = DATA.encode("utf-8")
params = "account_id=acct 2"
try:
    URL = "http://localhost:8080/lists?" + quote_plus(params)
    request = urllib.request.Request(url=URL,data=DATA,method=METHOD)
    response = urllib.request.urlopen(request)
...

I also have a request handler coded as follows (there are lot of print statements in here for debugging purposes):

class MyHandler(BaseHTTPRequestHandler):
...
def do_POST(self):
    length = int(self.headers['Content-Length'])
    print("HEADERS: ", self.headers)
    print (str(length))
    print(self.rfile)
    post_data = urllib.parse.parse_qs(self.rfile.read(length).decode('utf-8'))
    print(post_data)

This prints the following result to the console:

Starting thread
started httpserver...
HEADERS:  Accept-Encoding: identity
User-Agent: Python-urllib/3.3
Content-Length: 138
Content-Type: application/x-www-form-urlencoded
Host: localhost:8080
Connection: close


138
<_io.BufferedReader name=404>
{}

My questions:
1) In the server (do_POST), how do I access the data I think I'm sending with my request (i.e. {"listName":"Test list","listDesc":"A test...)?

2) Is my request even sending the data in the first place?

3) Is there a place where this is documented in novice-accessible terms?

Accra

You may find that using a web framework like Flask or Bottle will make this stuff much easier to do. – Kelp

@Kelp - Thanks for the advice. I tried using Bottle but it seemed really geared to using HTML which, in our case, we're not using. We're writing a RESTful application to balance work across multiple servers and it really doesn't have a browser component. Like almost all of these things, I also found Bottle's documentation to be great if you already knew your stuff but really hard for beginners who don't have a background in web concepts. – Accra

Answer 1

Give this a try. I stole it from an answer to another question

def do_POST(self):
    ctype, pdict = cgi.parse_header(self.headers.getheader('content-type'))
    if ctype == 'multipart/form-data':
        postvars = cgi.parse_multipart(self.rfile, pdict)
    elif ctype == 'application/x-www-form-urlencoded':
        length = int(self.headers.getheader('content-length'))
        postvars = cgi.parse_qs(self.rfile.read(length), keep_blank_values=1)
    else:
        postvars = {}

    print(postvars.get("listName", "didn't find it"))

Emprise

Thanks, Nate, this was really helpful. I ended up replacing the cgi.parse_qs with urllib.parse.parse_qs and it seems to work. I'm too tired to figure out why right now but the trick was the keep_blank_values=1 flag. – Accra

Answer 2

1) In the server (do_POST), how do I access the data I think I'm sending with my request (i.e. {"listName":"Test list","listDesc":"A test...)?

you can access the data just by:

print self.rfile.read(length).

after make sure this is working. you can do other parse work. I suggest use simplejson to decode the json string. urllib.parse.parse_qs seems unnecessary.

2) Is my request even sending the data in the first place?

the code looks fine. to make sure it works, just try:

    curl -d "asdf" http://yourhost:yourport

to see if the server have same response. 
so you can know whether the server side or client side goes wrong.

3) Is there a place where this is documented in novice-accessible terms?

the official document is always a good choice:
http://docs.python.org/2/library/basehttpserver.html

Sorcerer

For some reason, the print self.rfile.read(length) doesn't work for me - it just causes the python shell (IDLE in Win7) to hang. Also, the document you point to is a great case in point. I did pick up from it that I need to access the 'rfile' attribute but it doesn't say anything about the keep_blank_values parameter. I suspect there is some literature on the web that could help out but, because there is so much out there, finding the right item is tough. Anyhow, thanks for your help! – Accra