Python requests.get fails with 403 forbidden, even after using headers and Session object

Question

I'm making a GET request to fetch JSON, which works absolutely fine from any browser on any device, but not by python requests:

url = 'https://angel.co/autocomplete/new_tags'                         
params = {'query': 'sci', 'tag_type': 'MarketTag'}
resp = requests.get(url,params=params)
resp.raise_for_status()

gives HTTPError: 403 Client Error: Forbidden for url: https://angel.co/autocomplete/new_tags?query=ab&tag_type=MarketTag

So I tried:

1. Python requests. 403 Forbidden - I not only tried using User-Agent in headers but also all other headers that I found in Request Headers section in firefox for JSON response, but still 403!
2. Python requests - 403 forbidden - despite setting `User-Agent` headers - By making request through Session object, I still get 403!

What can be the possible cause? Is there something else I could try using?

EDIT: Request Headers (inspecting headers section of JSON in firefox) that I used in headers attribute:

{'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language':  'en-US,en;q=0.5',
'Connection': 'keep-alive',
'Host': 'angel.co',
'If-None-Match: 'W/"5857a9eac987138be074e7bdd4537df8"',
'TE': 'Trailers',
'Upgrade-Insecure-Requests': 1,
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0'}

Precipitation

please share all the headers that you have used – Gilcrest

@VikasSharma I've added the headers as an edit to my question. – Precipitation

I'm stuck with the exact same problem... could you resolve this issue somehow? – Scavenge

have the same issue. how di you solve this? – Theall

Answer 1

If a get request returns 403 Forbidden even after adding user-agent to headers, you may need to add more headers like this:

    headers = {
        'user-agent':"Mozilla/5.0 ...",
        'accept': '"text/html,application...',
        'referer': 'https://...',
    }
    r = requests.get(url, headers=headers)

In the chrome, Request headers can be found in the Network > Headers > Request-Headers of the Developer Tools. (Press F12 to toggle it.)

Heterosporous

doesnt seem to work...i think websites have found another way to prevent requests from code. – Theall

Answer 2

I assume you website detects, when a request isn' sent from a browser (made with javascript).

I had a similar issue recently, and this answer had worked for me.

Gogh