Python - make a POST request using Python 3 urllib

Question

I am trying to make a POST request to the following page: http://search.cpsa.ca/PhysicianSearch

In order to simulate clicking the 'Search' button without filling out any of the form, which adds data to the page. I got the POST header information by clicking on the button while looking at the network tab in Chrome Developer Tools. The reason I'm posting this instead of just copying solutions from the other similar problems is that I believe I may have not gotten the correct header information.

Is it properly formatted and did I grab the right information? I've never made a POST request before.

This is what I've managed to piece together:

import urllib.parse
import urllib.request


data = urllib.parse.urlencode({'Host': 'search.cpsa.ca', 'Connection': 'keep-alive', 'Content-Length': 23796,
                                     'Origin': 'http://search.cpsa.ca', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
                                     'Cahce-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest',
                                     'X-MicrosoftAjax': 'Delta=true', 'Accept': '*/*',
                                     'Referer': 'http://search.cpsa.ca/PhysicianSearch',
                                     'Accept-Encoding': 'gzip, deflate',
                                     'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6',
                                     'Cookie': 'ASP.NET_SessionId=kcwsgio3dchqjmyjtwue402c; _ga=GA1.2.412607756.1459536682; _gat=1'})


url = "http://www.musi-cal.com/cgi-bin/query?%s"

data = data.encode('ascii')
with urllib.request.urlopen("http://search.cpsa.ca/PhysicianSearch", data) as f:
    print(f.read().decode('utf-8'))

This solution outputs the page's HTML, but not with any of the data I wanted to retrieve from the POST request.

Dock

The POST data should correspond to the name attributes and values of the input elements of the form that you are submitting. You can pick up the name attributes by inspecting the html of the form. You're using the request headers as POST data - that won't work. Also, consider using requests package (docs.python-requests.org/en/master), it's friendlier than urllib2. – Baud

How do I specify the form? Or do I just need key-value pairs specifying the data? – Dock

In chrome, look at the POST request in the network tab like you did earlier and go to the bottom of the headers tab - there you will see the names and values whether it's a POST request or a GET request with query parameters – Baud

I did this, and a POST request is happening, but it's not actually grabbing the page. This is what requests.text gives me: 1|#||4|50|pageRedirect||%2fError.aspx%3faspxerrorpath%3d%2fPhysicianSearch| – Dock

Answer 1

This is how you do it.

from urllib import request, parse
data = parse.urlencode(<your data dict>).encode()
req =  request.Request(<your url>, data=data) # this will make the method "POST"
resp = request.urlopen(req)

Squiffy

What if I want to make a POST request with empty body? – Sontag

request.Request(..., method='POST'). docs.python.org/3/library/… – Flowing

It's recommended practice to do with request.urlopen(req) as resp: and do whatever you need to do with resp in that block. – Clitoris

Thank you Sir. You save my time. Your code working perfect on Python3 – Providence

So you mean the mistake in the question was the attempt to pass the URL and data to urllib.request.urlopen() directly? Where does the documentation say that that's wrong? – Felspar

@c panda, what can be a cause of 502 bad gateway error when I am using this piece of code when sending POST request to my API? – Hephzipah

Answer 2

Thank you C Panda. You really made it easy for me to learn this module.

I released the dictionary that we pass does not encode for me. I had to do a minor change -

from urllib import request, parse
import json

# Data dict
data = { 'test1': 10, 'test2': 20 }

# Dict to Json
# Difference is { "test":10, "test2":20 }
data = json.dumps(data)

# Convert to String
data = str(data)

# Convert string to byte
data = data.encode('utf-8')

# Post Method is invoked if data != None
req =  request.Request(<your url>, data=data)

# Response
resp = request.urlopen(req)

Dyl

Should really be a comment – Glomerulonephritis

if the service is strict in content type it accepts and it is json, then the following is also needed: req.add_header('Content-Type', 'application/json') https://mcmap.net/q/174164/-how-do-i-send-a-post-request-as-a-json – Bluh

You can combine the 3 manipulations you're doing to data in a single command: request.urlopen(url='your url', data=bytes(json.dumps(dict_obj), encoding='utf-8')) – Decrescent

For some reason, the accepted answer does not work with my FastAPI backend but this does. – Harlequinade

Answer 3

Set method="POST" in request.Request().

---

Sending a POST request without a body:

from urllib import request

req = request.Request('https://postman-echo.com/post', method="POST")
r = request.urlopen(req)
content = r.read()
print(content)

---

Sending a POST request with json body:

from urllib import request
import json

req = request.Request('https://postman-echo.com/post', method="POST")
req.add_header('Content-Type', 'application/json')
data = {
    "hello": "world"
}
data = json.dumps(data)
data = data.encode()
r = request.urlopen(req, data=data)
content = r.read()
print(content)

Antitoxic

My problem was that I was passing the data kwarg during the construction of request.Request, as in request.Request(..., data=...), where instead data should be passed to urllib.request.urlopen. – Tashinatashkent

this question should be in the top – Understrapper

Answer 4

The above code encoded the JSON string with some extra \" that caused me a lot of problems. This looks like a better way of doing it:

from urllib import request, parse

url = "http://www.example.com/page"

data = {'test1': 10, 'test2': 20}
data = parse.urlencode(data).encode()

req = request.Request(url, data=data)
response = request.urlopen(req)

print (response.read())

Golconda

Answer 5

It failed when I use urlencode. So I use the following code to make a POST call in Python3:

from urllib import request, parse

data = b'{"parameter1": "test1", "parameter2": "test2"}'
req = request.Request("http://www.musi-cal.com/cgi-bin/query?%s", data)
resp = request.urlopen(req).read().decode('utf-8')
print(resp)

Erotic