Google Trends Quota Limit

Question

I'm trying to pull data from Google trends and got a "You have reached your daily limit" error after only 2 tries.

Is there any way to go around this? I know Google API projects have special quota limits but Google Trends doesn't have an API. I also read that we may need to pass it a cookie file so that it seems like I'm logged in. Has anyone faced this issue before?

Hyperion

Answer 1

I'm struggling with the same issue! From your question I can't figure out what stage have you achieved... But here is the solution that I've found:

1. You should emulate browser with cookies. I think the best way to do it is to use Mechanize library.
2. At first your program should "login" using GET request to "https://accounts.google.com/Login?hl=en"
3. Immediately after that you can access some other personal resources, but not google trends!
4. After some significant time you can successfully get google trends data as CSV.
5. I still have not discovered the exact time period, but it is more than 10 minutes and less than several hours :). That is why saving your cookies for latter use is a good idea!

Few more tips:

• If you are developing using python / ruby under Windows do not forget to set up CA ROOT certificates package for OpenSSL library. Otherwise HTTPS connection will fail and you won't login! See Getting the `certificate verify failed (OpenSSL::SSL::SSLError)` erro with Mechanize object

• I recommend you to save cookies to external file at program shutdown. And restoring them at startup.

• Do not forget to allow redirects, because Google is using redirects all the time.

Ruby code example:

require 'mechanize'
require 'logger'
begin
  agent = Mechanize.new { |a|
    a.user_agent = 'Opera/9.80 (Windows NT 5.1) Presto/2.12.388 Version/12.16'

    cert_store = OpenSSL::X509::Store.new
    cert_store.add_file 'cacert.pem'
    a.cert_store = cert_store

    a.log = Logger.new('mech.log')

    if File.file?('mech.cookies')
      cookies = Mechanize::CookieJar.new
      cookies.load('mech.cookies')
      a.cookie_jar = cookies
    end

    a.open_timeout = 5
    a.read_timeout = 6
    a.keep_alive   = true
    a.redirect_ok  = true
  }

  LOGIN_URL = "https://accounts.google.com/Login?hl=en&continue=http://www.google.com/trends/"
  login_page = agent.get(LOGIN_URL)
  login_form = login_page.forms.first
  login_form.Email = *
  login_form.Passwd = *
  login_response_page = agent.submit(login_form)

  page = agent.get(url)

  # DO SOME TRENDS REQUESTS AFTER SIGNIFICANT PERIOD OF TIME

ensure
  if agent
    agent.cookie_jar.save('mech.cookies')
  end
end

Boyla

Answer 2

You probably disabled your cookies, which makes Google Trends think you're a robot

Noria

Worked for me - I'd left adblock turned on which did it. Disable it on the domain and it's fine. – Strew

Answer 3

I think I have found a way to solve the problem. Just make sure that you call the Google Trends API with the cookie PREF. That is you don't need to login the Google account. Of course, you don't need to emulate browser. The cookie PREF is just enough.

OK. Where the cookie PREF comes from? It is very easy. Just open the browser, and login in your Google account. Finally, look up the cookie PREF under the Google website, it is just under the domain www.google.com.Then copy the value of the cookie PREF to your program or script. That's all.

I have called the Google Trends API hundreds of times in several seconds by this way. Good Luck to you!

Marketplace

can you tell me how to use it in java?? – Overlook

I used the cookie NID and it worked for me for at least 30 requests in a minute, for at least 5 minutes (haven't stress tested yet) – Rheumy

Answer 4

I found this paper about prevention or just a Zeta-Jones effect in google Trends, it was so useable: G Fond, A Gamanb, E Haffenb, P Llorca. "Google Trends: ready for real-time suicide prevention or just a Zeta-Jones effect ?." International Journal of Computer Networks and Communications Security 3, no. 1 (2015): 1-5.

Colchicum

Can you explain how this answers the OP's question? – Phosphoprotein