Works with urrlib.request but doesn't work with requests

Question

I am trying to send a request wtih post method to an API, my code looks like the following one:

import urllib.request
import json

url         = "https://api.cloudflareclient.com/v0a745/reg"
referrer    = "e7b507ed-5256-4bfc-8f17-2652d3f0851f"
body        = {"referrer": referrer}
data        = json.dumps(body).encode('utf8')
headers     = {'User-Agent': 'okhttp/3.12.1'}
req         = urllib.request.Request(url, data, headers)
response    = urllib.request.urlopen(req)
status_code = response.getcode()
print (status_code)

Actually it works fine but i want to use "requests" library instead as it's more updated and more flexible with proxies with following code:

import requests
import json

url         = "https://api.cloudflareclient.com/v0a745/reg"
referrer    = "e7b507ed-5256-4bfc-8f17-2652d3f0851f"
data        = {"referrer": referrer}
headers     = {'User-Agent': 'okhttp/3.12.1'}
req         = requests.post(url, headers=headers, json=data)
status_code = req.status_code
print (status_code)

But it returns 403 status code, how can i fix it ?

Keep in mind that this API is open to everyone and you can just run the code with no worries.

---

EDIT-1: i have tried removing json.dumps(body).encode('utf8') or just .encode('utf8') from the second code by @tomasz-wojcik advice but i am still getting 403 while the first code still works!

EDIT-2: i tried requesting with postman that successfully made the request and returned 200 status code. postman generated the following python code:

import requests

url = "https://api.cloudflareclient.com/v0a745/reg"

payload = "{\"referrer\": \"e7b507ed-5256-4bfc-8f17-2652d3f0851f\"}"
headers = {
  'Content-Type': 'application/x-www-form-urlencoded',
  'User-Agent': 'okhttp/3.12.1',
  'Host': 'api.cloudflareclient.com'
}

response = requests.request("POST", url, headers=headers, data=payload)

status_code = response.status_code
print (status_code)

If you run the code outside of postman, it still returns 403 status code, i'm a litte confused, i am thinking that maybe "requests" library doesn't changing the user-agent in the second code.

EDIT-3: I have looked into it and found out that it works on python 2.7.16 but doesn't work on python 3.8.5!

EDIT-4: Some Developers are reporting that the second code works on python 3.6 too but the main thing is why it is working on other versions but not working on 3.8 or 3.7 ?

Python Versions that returned 403 status code(second code): 3.8.5 & 3.7

Python Versions that returned 200 status code(second code): 3.6 & 2.7.16

Gorblimey

your data in requests is different from the first one – Ayannaaycock

For requests body, the library cannot help you, you have to take care of it – Ayannaaycock

What is the status code with the urllib? Have you used all the same url for the both libs? Is there correct protocol prefix (http:// or https://) in case od requests? – Swordfish

The urllib code returns 200 status code! They both have the same url assigned to and both have (https://) in prefix of url! @AndrejZacharevicz – Gorblimey

Ty for mentioning this, I have changed my code but the problem still exists and i'm getting 403 status code! @Ayannaaycock – Gorblimey

What's wrong with postman? – Theism

Nothing wrong with it, just i am implementing the code not using postman, that is just to clarify that the postman works fine with the request but postman generated code doesn't and it's written in "requests" library. – Gorblimey

@Gorblimey I would say it's difficult to debug without website name and other information that you are passing along – Ayannaaycock

@Ayannaaycock everything is described now including API url that as i mentioned is open to everyone and you can run both codes with no worries aswell. – Gorblimey

@Gorblimey I got 403 status code when I run the first code. – Tenorrhaphy

Are you sure you mean urllib code ? cause i tested it on separate systems with different os and ips! – Gorblimey

Yes, I'm sure. I suspect that it is because I don't know "some referrer code". – Tenorrhaphy

@Asocia i have changed the "some referrer code" to exact value in my question although the last value was working for like 10 different systems. – Gorblimey

I get 403 for all code examples. – Karakalpak

@PedroRodrigues what is your os and requests or urrlib versions ? – Gorblimey

Using Google Colab's Notebook colab.research.google.com/drive/… – Karakalpak

@PedroRodrigues Maybe it's just not changing the user-agent as it sends the request through a webservice or browser. – Gorblimey

@Gorblimey All your requests examples are giving 403 – Ayannaaycock

@Ayannaaycock There is a lot of guys confirming that they are getting 200, please describe your operating system and python version so i can debug the problem better. – Gorblimey

@Gorblimey MacOS, python 3.7 – Ayannaaycock

@Ayannaaycock We had a report that the second code worked on python 3.6, Also i can confirm that second code runs on python 2.7.16 with 200 status, in the meantime python 3.8.3 runs the first code with 200 status code. – Gorblimey

Answer 1

The issue seems to be with how the host is handling ssl. Newer versions of requests uses certifi which in your case is having issues with the host server. I downgraded requests to an earlier version and it worked. (2.1.0). You can fix the version in your requirements.txt and it should work with any python version.

https://requests.readthedocs.io/en/master/user/advanced/#ca-certificates

Before version 2.16, Requests bundled a set of root CAs that it trusted, sourced from the Mozilla trust store. 
The certificates were only updated once for each Requests version. When certifi was not installed, this led to extremely out-of-date certificate bundles when using significantly older versions of Requests.

For the sake of security we recommend upgrading certifi frequently!

Sinkage

Thank you, this is working, you mean the host rejects the new set of certificates included in newer versions of requests ? – Gorblimey