I am getting a different response from python and curl, although each uses the exact same parameters.
Python:
import requests
headers = {
'Accept-Language': 'en-US,en',
'Accept': 'text/html,application/xhtml+xml,application/xml',
'Authority': 'www.google.com',
'User-Agent': 'SomeAgent',
'Upgrade-Insecure-Requests': '1',
}
response = requests.get('https://www.avvo.com', headers=headers)
# Returns a 403 response
Curl:
import shlex, subprocess
cmd = '''curl -H 'Accept-Language: en-US,en' -H 'Accept: text/html,application/xhtml+xml,application/xml' -H 'Authority: www.google.com' -H 'User-Agent: SomeAgent' -H 'Upgrade-Insecure-Requests: 1' https://www.avvo.com'''
args = shlex.split(cmd)
process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
# Returns a 200 response
Both requests are being sent from the same IP. It looks like it's a cloudflare issue, is there any way cloudflare can detect a request coming from the python requests library vs a direct curl command?
I left the website in the code in case its useful to run. Here is the curl command directly:
curl -H 'Accept-Language: en-US,en' -H 'Accept: text/html,application/xhtml+xml,application/xml' -H 'Authority: www.google.com' -H 'User-Agent: SomeAgent' -H 'Upgrade-Insecure-Requests: 1' https://www.avvo.com/administrative-law-lawyer/ny.html