Permission denied error when calling Google Cloud Run API from Cloud Scheduler

Question

The following setup exist and works fine:

• Google Cloud Run (my API worked when I used the open URL - i.e. something like this https://some_name-ab3adasmjk-oa.a.run.app)

• Google Cloud Scheduler (calling the above API by a HTTP, something like this URL = https://some_name-ab3adasmjk-oa.a.run.app/v1/myapi?id=0)

Now the problem: I PRESSED THIS BUTTON "SECURITY" that was shown inside my Google Cloud Run console!

After adding the "SECRET MANAGER SECRET ACCESSOR" to the Service-Account of the Google Cloud Run service (as required after pressing this "Security" button), then the Scheduler no longer works!

I followed all steps that are described in this post - but the "permission denied" for the Scheduler still remains!

What I did:

A) Add "SECRET MANAGER SECRET ACCESSOR" to the Service Account of the Google Cloud Run (as expected after pressing this button).

B) Add "OIDC Token" inside the Google Cloud Scheduler

C) Also, the Google Cloud Run API is hosted on a different "region" than the Google Cloud Scheduler. Is that a problem? (It was no problem yesterday!)

Did I miss something?

Questions:

1. After the service-account has the "SECRET MANAGER SECRET ACCESSOR", do I still use the Google Cloud Run URL inside the Scheduler - or do I need a different URL now? (https://some_name-ab3adasmjk-oa.a.run.app ?)

2. Inside the Scheduler, I did add "Add OIDC Token" and giving the Service Account of my Cloud Run API. I left Audience blank. Is that correct? Or is something else missing?

---

Additional information:

Steps I did and questions:

0.) Is it a problem to be on two different regions for Cloud Run API and Cloud Scheduler?

I) I created a first Service Account SA1 for the Google Cloud Run, giving it the SECRET MANAGER SECRET ACCESSOR permission

II) As mentioned in post, I created a second Service Account SA2 to IAM: i.e. [e-mail] giving it the role Cloud Scheduler Service Agent

III) Is it correct that under Add OIDC token, I need to add SA1? (or is it SA2?)

IV) Not sure how this works: As desribed in the same post, there is the need to grant roles/cloudfunctions.invoker (or if using a 2nd Generation Environment Server, then roles/run.invoker instead)

--> how do I do that exactly? Step-by-step description PLEASE! --> What is the exact naming of the Permission if I want to grant Access? All I can find in the Web-Console is Clud Run Service Agent - but where can I find the option roles/run.invoker?

--> To what Service Account do I have to give this roles/run.invoker permission ? (i.e. is it to SA1 or SA2?)

Peloquin

What did you specify for the OIDC audience? If not specified, by default, the entire URL is used as the audience (including request parameters). The audience must match the Cloud Run service URL. – Eastbound

Good Catch John, it was my question! The audience must be the Cloud Run service audience. If you use a custom audience on Cloud Run, you should use it also – Jolin

Thanks, Guillaume. To clarify, use this audience https://some_name-ab3adasmjk-oa.a.run.app and not https://some_name-ab3adasmjk-oa.a.run.app/v1/myapi?id=0. – Eastbound

I added the Audience, but it still does not work. Please see Additional Information on the original post with more questions. – Peloquin

The service account that Cloud Scheduler is using needs Cloud Run Invoker. That is the service identity that is calling Cloud Run. See this link. You can test various scenarios using gcloud auth print-identity-token and the CLI tool curl. cloud.google.com/sdk/gcloud/reference/auth/print-identity-token and cloud.google.com/run/docs/authenticating/developers#curl – Eastbound

I achieved setting roles/run.invoker successfully via CLI (..did not know it existed ;)). I also placed the Scheduler on the same region as the Cloud Run. BUT IT STILL DOES NOT WORK. This is turning me nuts! – Peloquin

The first step is to get organized. Review what you have configured, read the requirements from the documentation, and then break the problem into pieces. First, disable authorization on Cloud Run and test the service with the public URL. once that works, enable authorization and test using the CLI curl as I showed previously. Once that works, debug Cloud Scheduler calling your Cloud Run endpoint. – Eastbound

NOW IT WORKS !!!! I had to delete the Cloud Run completely! For anybody also confused: roles/run.invoker is the glc-terminal command - and if you use the console, it is called Cloud Run Invoker permission. The confusion not enough, I give a detailed explanation as an answer here below. Thank you @JohnHanley for your kind support! – Peloquin

Answer 1

huuhh and another finding:

If you use the Console to add an Auth-Header (OIDC token with Cloud Run Service and Audience-URL of the Cloud Run-baseURL), then please don't forget to press "CONTINUE" button before you press the "UPDATE" butten !!!!

(i.e. the UPDATE-button is so close that one feels tempted to immediately update. But if you do so, then for some odd reason the scheduler does not add Auth-Header and you end up with 403 permission-denied error)

Peloquin

3 days of trial and error and this resolved it for us. Great find. All I had to do was click through this and our permission denied issue was resolved. No changes to configuration. – Mathison

Answer 2

1. Creater a first Service Account SA1 for Cloud Run with role-permission Cloud Run Invoker All you need is to create a new Service Account under IAM->Service Account find the button Create Service Account and create one. Keep its email (is.e something like this [someName]@[PROJECT_ID].iam.gserviceaccount.com

2. Go to the Cloud Run side-panel, Add principal and under New principal, enter this email. Then pick the role Cloud Run Invoker !

3. Create a second Service Account SA2 for Scheduler with role-permission Cloud Scheduler Service Agent - see link. All you need is the Scheduler Service Email that looks something like that: service-[PROJECT_NUMBER]@gcp-sa-cloudscheduler.iam.gserviceaccount.com

4. Inside your Cloud Scheduler, make sure to add OIDC Token and giving the Service Account of your Cloud Run API (i.e. [someName])

5. Make sure you enter the Audience URL = URL from your Cloud Run (i.e. something like https://some_name-ab3adasmjk-oa.a.run.app)

6. Make sure you set Require authentication inside your Cloud Run.

To sort out some confusion:

roles/run.invoker is the glc-terminal command - and if you use the console, it is called Cloud Run Invoker permission. The two are equivalent!

What almost turned me nuts is the fact that this Cloud Run Invoker permission can only be found when you have the Cloud Run console open and select your service. After selecting, you find permission menu on the right-hand panel and can address the Cloud Run Invoker with the "Add principal" as described in step 2 above.

But what absolutely does not work: You will never be able to set Cloud Run Invoker inside IAM --> Service Account --> Permissions. It just does not show up. That's actually what made me most crazy. The Cloud Run Invoker permission on the console only exist if you start the process on the right-hand-side panel of your Cloud Run - AND ONLY IF YOU SELECT ONE CLOUD RUN - otherwise you never find it!

I guess you should use glc-terminal cmds anyways since the console is so hidous !!!

Peloquin

Your problem arises due to confusion on what an identity is and what a resource is. When you tried to modify the service account, the service account is the resource. That resource only supports service account permissions that you grant to another identity (to use the service account). The key is to use the service account as an identity and modify who it calls. In your case Cloud Run. Cloud Run is the resource so you modify its policy to allow the service account's identity to invoke the Cloud Run instance. The key is to understand identities and resources. – Eastbound