403 Request rate higher than configured

Question

When I'm adding members to Google Groups I'm receiving a 403 Request rate higher than configured error. This is happening for all applications, including the API Explorer on the Google developer website.

Even if I make only a single request in 12+ hours I still receive the error. I've confirmed the Admin API has plenty of quota remaining on the project.

It is happening only for a single Apps for Education domain.

Request:

POST https://www.googleapis.com/admin/directory/v1/groups/[redacted]%40ryan.catholic.edu.au/members?key={YOUR_API_KEY}
{
 "email": "<redacted>",
 "role": "MEMBER"
}

Response:

{
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "quotaExceeded",
    "message": "Request rate higher than configured."
   }
  ],
  "code": 403,
  "message": "Request rate higher than configured."
 }
}

Primogenitor

The above error is thrown when limit of concurrent requests for a certain operation has been reached. You need to slow down the requests, this can be done by implementing exponential back off. check this link developers.google.com/drive/web/handle-errors – Cuticula

I realise that. However it happens on a single request over 12 hours after the last request. It is also affecting all applications, including the API explorer in Google. – Primogenitor

@Cuticula I'm running round in circles with support on this. If the issue is reproducible through the API explorer should I assume it's a problem at Google's end? My ticket is #06532690 – Primogenitor

Answer 1

We have a partial resolution to this.

Issue Summary:

• API returns '403 Request rate higher than configured' on certain requests, even after no requests are made for a long period of time (12+hrs) to a single domain.
• Attempting to use a different Client ID or and user return the same error code. It also impacts the API Explorer.

Cause:

• Certain operations can trigger a "black-listing" of a domain (not an application) for abuse. There may be undocumented limits on specific operations within a time period.

Resolution:

• Support receive a notification of this blacklisting and can remove the flag.

---

In our case we were likely adding too many members to too many groups in a 24 hour period and triggered an undocumented limit (support couldn't tell me what the limit was). We're still waiting for the black-listing to be removed from our domain.

This problem does raise additional questions however:

• The API limit documentation https://developers.google.com/admin-sdk/directory/v1/limits does not appear to be complete. There appears to be additional limits that can trigger a "domain" black-list. What are these limits?
• Why does a Google Apps Domain become black-listed rather than the application making the requests?
• Why aren't the Domain Administrator's notified of a black-listing of their domain?

Primogenitor

Unfortunately, while support can see the blacklisting, they don't seem to be able to remove it. Still working with support to try and fix it, but it's been almost a month now with no resolution :( – Primogenitor

Peter, do you have some more context about this? Specifically where did you get this information and in what context. Would like to look into this. Cheers. – Galactopoietic

Answer 2

We just hit this 3/16/16, does anyone know if this has been resolved? Should I immediately engage support or do they correctly remove the blacklisting automatically now after a time period.

That will teach me not to run two syncs, even with limited changes within 30 minutes of each other.

I know this is an old thread but it seems that others might still be experiencing this and advice is appreciated.

Targum

This is not an answer to the original question and shouldn't be posted as such. This isn't a discussion thread like that. There are guidelines on how to use the stackoverflow.com/help/how-to-answer. If you have a new question relating to this then start a new question stackoverflow.com/help/how-to-ask – Allhallowmas

I would engage support immediately. We disabled syncs for almost a week, and that wasn't enough to remove the blacklisting. – Primogenitor