Raised Google Drive API Per-user limit, still getting userRateLimitExceeded errors

Question

Similar to Raising Google Drive API per-user limit does not prevent rate limit exceptions

In Drive API Console, quotas looks like this:

Despite Per-user limit being set to an unnecessarily high requests/sec, I am still getting rate errors at the user-level.

What I'm doing:

I am using approx 8 threads uploading to Drive, and they are ALL implementing a robust exponential back-off of 1, 2, 4, 8, 16, 32, 64 sec back-off respectively (pretty excessive back-off, but necessary imho). The problem can still persists through all of this back-off in some of the threads.

Is there some other rate that is not being advertised / cannot be set?

I'm nowhere near the requests/sec, and still have 99.53% total quota. Why am I still getting userRateLimitExceeded errors?

Ornas

Answer 1

userRateLimitExceeded is flood protection basically. Its used to prevent people from sending to many requests to fast.

Indicates that the user rate limit has been exceeded. The maximum rate limit is 10 qps per IP address. The default value set in Google Developers Console is 1 qps per IP address. You can increase this limit in the Google Developers Console to a maximum of 10 qps.

You need to slow your code down, by implementing Exponential Backoff.

1. Make a request to the API
2. Receive an error response that has a retry-able error code
3. Wait 1s + random_number_milliseconds seconds
4. Retry request
5. Receive an error response that has a retry-able error code
6. Wait 2s + random_number_milliseconds seconds
7. Retry request
8. Receive an error response that has a retry-able error code
9. Wait 4s + random_number_milliseconds seconds
10. Retry request
11. Receive an error response that has a retry-able error code
12. Wait 8s + random_number_milliseconds seconds
13. Retry request
14. Receive an error response that has a retry-able error code
15. Wait 16s + random_number_milliseconds seconds
16. Retry request
17. If you still get an error, stop and log the error.

The idea is that every time you see that error you wait a few seconds then try and send it again. If you get the error again you wait a little longer.

Quota user:

Now I am not sure how your application works but, If all the quests are coming from the same IP this could cause your issue. As you can see by the Quota you get 10 requests per second / per user. How does Google know its a user? They look at the IP address. If all your reuqests are coming from the same IP then its one user and you are locked to 10 requests per second.

You can get around this by adding QuotaUser to your request.

quotaUser - Alternative to userIp. Link

1. Lets you enforce per-user quotas from a server-side application even in cases when the user's IP address is unknown. This can occur, for example, with applications that run cron jobs on App Engine on a user's behalf.
2. You can choose any arbitrary string that uniquely identifies a user, but it is limited to 40 characters.
3. Overrides userIp if both are provided.
4. Learn more about capping usage.

If you send a different quotauser on every reqest, say a random number, then Google thinks its a different user and will assume that its only one request in the 10 seconds. Its a little trick to get around the ip limitation when running server applications that request everything from the same IP.

Squamulose

In my edit I describe the exponential back-off I'm aready implementing. For my app's architecture: most recently there were 4 threads from one IP making the requests. These threads have to do local IO, call other web services, and lastly upload files to Drive -- they are not only requesting Drive the whole time, making it unlikely 4 threads are breaking 10 requests per sec. --> even with this architecture, I was getting userRateLimitExceededErrors and curiously enough I was not before, when I happened to be using 8 threads per IP – Ornas

8 threads running the same ip could send 8 requests a second. chances that one or two run just fast enough to kick out the 10 is very high. Just add quotauser to your request it will fix the problem instantly. – Squamulose

your quotaUser work around sounds very compelling, if I'm understanding you correctly. Why would Drive check IP for "user" rather than the Drive user being used / impersonated? – Ornas

you have a request url going to google now. add this on the end &quotaUser=12345 (12345 should be a random number) All it does is tell Google that this is a different user then the prevous call you made. Your quota is 10 requests per second per "USER" if every thread call is a different user problem solved. – Squamulose

you could always run your threads and log when each is called to see if it is in fact what I say flood protection. – Squamulose

why is that error response page not part of Drive docs? Are they re-organizing documentation? (I see v3 in url) – Ornas

Because it's a general error with all of the APIs. Its not specific to drive. – Squamulose

updapte: This seemed to be working for a while... but now it's not quite. I've been adding random unique (UUID) value for quotaUser to every request, but am getting 403 userRateLimitExceeded on file insert. This is from 15 threads running from a single IP. Doesn't seem to be the silver bullet. Any ideas? – Ornas

would starting a second API project, and creating another Drive service account within that API project, avoid all these errors / increase speed? – Ornas

I don't think you are going to ever completely remove this error. Just keep retrying slowing it down a little. Once it runs again speed it back up. I have seen this with the Google Analytics API, ripping 1 million rows can take a long time. The server wont let you go to fast. – Squamulose

i just don't understand something, and my boss is looking for answers -- if Google is so large, they ought to have 1000's of servers which they can load balance all my (and others) Drive API requests among. Yet I'm often getting HTTP time-out errors from them (along with API erros) -- how can this be? – Ornas

You aren't the only dev using it they have to be fair and set a limit for us all. – Squamulose