Why google retrieves different info to scrapers
Asked Answered
C

0

7

I have been working with pytrends, a package to retrieve google trends data, a long while now and realised that the results I get on the browser and using pytrends differ quite a bit. After checking the request each are doing, the only difference I was able to spot was the parameter user type inside the request they both made, and some minor changes, browser makes the request indicating timezone twice, for example.

Browser:
"userConfig":{"userType":"USER_TYPE_LEGIT_USER"}
Pytrends:
"userConfig": {"userType": "USER_TYPE_SCRAPER"}

Timeframes, timezones, and the rest of parameters in the requests are the same but the token, which you have to get before doing the actual request for the data. I don't know why this may be happening, and I don't feel the token has anything to do, as you request it, again, with the only difference in both request of specifying different user types

Now I post the two full request each make but the token:

Pytrends

https://trends.google.com/trends/api/widgetdata/multiline?req={"time": "2014-12-28 2020-01-01", "resolution": "WEEK", "locale": "es", "comparisonItem": [{"geo": {"region": "ES-CM"}, "complexKeywordsRestriction": {"keyword": [{"type": "BROAD", "value": "gripe"}]}}], "requestOptions": {"property": "", "backend": "IZG", "category": 0}, "userConfig": {"userType": "USER_TYPE_SCRAPER"}}&token=TOKEN_HERE&tz=-120
Browser
https://trends.google.es/trends/api/widgetdata/multiline?hl=es&tz=-120&tz=-120&req={"time":"2014-12-28 2020-01-01","resolution":"WEEK","locale":"es","comparisonItem":[{"geo":{"region":"ES-CM"},"complexKeywordsRestriction":{"keyword":[{"type":"BROAD","value":"gripe"}]}}],"requestOptions":{"property":"","backend":"IZG","category":0},"userConfig":{"userType":"USER_TYPE_LEGIT_USER"}}&token=TOKEN_HERE

Does anyone know why is this happening and how can I manage to retrieve consistent data between both of them? Also, why does Google give different data if you are scraping their website rather than using a browser?

Closestool answered 7/10, 2022 at 13:51 Comment(4)
Google may remember all your visted pages in browser and use this to send different content.Thermos
Not quite, results are the same even in a different computer with a different account. Only change produced when it detects data is being scraped.Closestool
Technically speaking, pytrends is a scraper, so the the userType does seems correct.Gould
Yeah, but that's the thing, not that it shouldn't be detected as a scraper, but that it retrieves different info for scrapers. Why? Which info is correct? Are both just different samples of the total of searches?Closestool

© 2022 - 2025 — McMap. All rights reserved.