GCP IAP There was a problem with your request. Error code 11

Question

I configured Google Cloud IAP for AppEngine, and then, when I try to access my app, I login to my google account, and I am redirected to this url:

project-id.appspot.com/_gcp_gatekeeper/authenticate?state=long-state-string&scope=email%20openid%20https://www.googleapis.com/auth/userinfo.email&authuser=2&hd=domain.com&session_state=a23b09e8ae90697d931d96068fdcf163a7a71010..9268&prompt=consent

This page just shows the following error, and nothing else matters:

There was a problem with your request. Error code 11

Tried looking for logs everywhere but in vain. @Matthew Sachs Do you have any idea ?

Disputatious

Answer 1

Experienced this when setting up a new project and the solution was to turn IAP off and then back on again.

Not a great answer but hopefully will help.

Zirconia

Hopefully we will get a better answer! – Zirconia

Glad you were able to get this resolved! The one case I've seen this in is when the client ID and client secret that IAP is configured with are wrong. If this happens again, I'd be curious to know whether the values shown in cloud.google.com/appengine/docs/admin-api/reference/rest/v1/… match the client ID and secret in console.cloud.google.com/apis/credentials . To check the client secret v. IAP's "sha256" hash, you can use a tool like echo -n secret | sha256sum, or Python pastebin.com/f9EUKECL – Ungrudging

Will update this and get those values for you if we encounter it again. Thanks Matthew. – Zirconia

Thank you so much! I think that if ssl cert has changed then IAP need refresh (turn off and on). – Direction

Answer 2

Just ran into this and have a theory about cause and additional details about solution.

giotr's theory of cause has merit:
Ran into this issue in July 24th, reset worked, then ran into it again in Sept 12th. Notice that's ~2-3 months / about the time it takes for an ephemeral ACME provisioned HTTPS cert to rotate before expiring. I'm guessing there's some logic that occasionally glitches up where the cert rotating can cause an issue due to imperfect software logic that results in a semi rare bug with IAP that requires a reset to solve.

More details about the fix:

• Turn off and back on can fail if you do it too quickly.
• Try turn off, wait 5 min, verify no longer protected by IAP, turn back on, wait 5 min, retest.
• If that doesn't work another thing to try is. Identify the backend service. Reboot backend service. Turn IAP off, wait 5 min, verify no longer protected by IAP, turn back on, wait 5 min, retest.

Update (possibly found root cause):

• The fix still works, but the theorized root cause seems to be incorrect, as it happened again within after 2 weeks, so the theory of HTTPS cert refresh stops making sense.
• New observation lead to a new theory. 6 environments were protected by IAP, but 5 of 6 environments would work perfectly / never break. 1 of 6 environments would periodically break randomly without cause. Noticed an important difference in the 1 environment that kept breaking. It's IAP ID/Secret stored in a kube secret (the kube secret was sourced from a kube software bot called "external secrets operator", that fetched it from GCP secrets manager which might not be relevant, but worth mentioning), had a new line at the end of the secret. The other 5 environments that were bulletproof (as in never broke) had no new line at the end of the secrets contents.
• So the point is if you're running into inconsistent odd breakage, check if your kube secret has a new line at the end of the secret, and remove the new line, before doing the fix, that newline leading to inconsistency may be the root cause.

Burley

Just ran into this again too soon for it to be explainable by cert rotation. (July 21st, Sept 15th, then Oct 2nd.) If this bugs you, and the following is an option, consider trying GKE's Gateway API Controller, it might be more algorithmically correct / dev work seems to be going into it and not GKE Ingress. – Burley

A bug is opened to address the issue to Google: issuetracker.google.com/issues/303097660 – Posehn