Cannot invoke Google Cloud Function from GCP Scheduler

Question

I've been trying to invoke a GCP function (--runtime nodejs8 --trigger-http) from GCP scheduler, both located within the same project. I can only make it work, if I grant unauthenticated access by adding the allUsers member to the functions permissions, with the Cloud Functions-Invoker role applied to it. However, when I only use the service account of the scheduler as the Cloud Functions-Invoker, I get a PERMISSION DENIED Error.

I created a hello world example, to show in detail, how my setup looks like.

1. I set up a service account:

gcloud iam service-accounts create scheduler --display-name="Task Schedule Runner"

2. Setting the role:

svc_policy.json:
{
    "bindings": [
      {
        "members": [
          "serviceAccount:[email protected]"
        ],
        "role": "roles/cloudscheduler.serviceAgent"
      }    
    ]
  }

gcloud iam service-accounts set-iam-policy [email protected] svc_policy.json  -q

3. Deploying the Cloud Function:

gcloud functions deploy helloworld --runtime nodejs8 --trigger-http --entry-point=helloWorld

4. Adding the service account as a member to the function:

gcloud functions add-iam-policy-binding helloworld --member serviceAccount:[email protected] --role roles/cloudfunctions.invoker

5. Creating the scheduler job:

gcloud beta scheduler jobs create http test-job --schedule "5 * * * *" --http-method=GET --uri=https://us-central1-mwsdata-1544225920485.cloudfunctions.net/helloworld --oidc-service-account-email=[e-mail] --oidc-token-audience=https://us-central1-mwsdata-1544225920485.cloudfunctions.net/helloworld

Log: PERMISSION DENIED

{
 httpRequest: {
 }
 insertId: "1ny5xuxf69w0ck"  
 jsonPayload: {
  @type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"   
  jobName: "projects/mwsdata-1544225920485/locations/europe-west1/jobs/test-job"   
  status: "PERMISSION_DENIED"   
  targetType: "HTTP"   
  url: "https://us-central1-mwsdata-1544225920485.cloudfunctions.net/helloworld"   
 }
 logName: "projects/mwsdata-1544225920485/logs/cloudscheduler.googleapis.com%2Fexecutions"  
 receiveTimestamp: "2020-02-04T22:05:05.248707989Z"  
 resource: {
  labels: {
   job_id: "test-job"    
   location: "europe-west1"    
   project_id: "mwsdata-1544225920485"    
  }
  type: "cloud_scheduler_job"   
 }
 severity: "ERROR"  
 timestamp: "2020-02-04T22:05:05.248707989Z"  
}

Update

Here are the corresponding settings.

Scheduler Service Account

gcloud iam service-accounts get-iam-policy [email protected]

bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudscheduler.serviceAgent
etag: BwWdxuiGNv4=
version: 1

IAM Policy of the function:

gcloud functions get-iam-policy helloworld    
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudfunctions.invoker
etag: BwWdxyDGOAY=
version: 1

Function Description

gcloud functions describe helloworld
availableMemoryMb: 256
entryPoint: helloWorld
httpsTrigger:
  url: https://us-central1-mwsdata-1544225920485.cloudfunctions.net/helloworld
ingressSettings: ALLOW_ALL
labels:
  deployment-tool: cli-gcloud
name: projects/mwsdata-1544225920485/locations/us-central1/functions/helloworld
runtime: nodejs8
serviceAccountEmail: [email protected]
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-671641e6-3f1b-41a1-9ac1-558224a1638a/b4a0e407-69b9-4f3d-a00d-7543ac33e013.zip?GoogleAccessId=[e-mail]&Expires=1580854835&Signature=S605ODVtOpnU4LIoRT2MnU4OQN3PqhpR0u2CjgcpRcZZUXstQ5kC%2F1rT6Lv2SusvUpBrCcU34Og2hK1QZ3dOPluzhq9cXEvg5MX1MMDyC5Y%2F7KGTibnV4ztFwrVMlZNTj5N%2FzTQn8a65T%2FwPBNUJWK0KrIUue3GemOQZ4l4fCf9v4a9h6MMjetLPCTLQ1BkyFUHrVnO312YDjSC3Ck7Le8OiXb7a%2BwXjTDtbawR20NZWfgCCVvL6iM9mDZSaVAYDzZ6l07eXHXPZfrEGgkn7vXN2ovMF%2BNGvwHvTx7pmur1yQaLM4vRRprjsnErU%2F3p4JO3tlbbFEf%2B69Wd9dyIKVA%3D%3D
status: ACTIVE
timeout: 60s
updateTime: '2020-02-04T21:51:15Z'
versionId: '1'

Scheduler Job Description

gcloud scheduler jobs describe test-job
attemptDeadline: 180s
httpTarget:
  headers:
    User-Agent: Google-Cloud-Scheduler
  httpMethod: GET
  oidcToken:
    audience: https://us-central1-mwsdata-1544225920485.cloudfunctions.net/helloworld
    serviceAccountEmail: [email protected]
  uri: https://us-central1-mwsdata-1544225920485.cloudfunctions.net/helloworld
lastAttemptTime: '2020-02-05T09:05:00.054111Z'
name: projects/mwsdata-1544225920485/locations/europe-west1/jobs/test-job
retryConfig:
  maxBackoffDuration: 3600s
  maxDoublings: 16
  maxRetryDuration: 0s
  minBackoffDuration: 5s
schedule: 5 * * * *
scheduleTime: '2020-02-05T10:05:00.085854Z'
state: ENABLED
status:
  code: 7
timeZone: Etc/UTC
userUpdateTime: '2020-02-04T22:02:31Z'

Colobus

cloud.google.com/functions/docs/securing/authenticating – Intestine

I did all that. The only official tutorial provided by GCP only deals with scheduler -> PubSub -> Cloud Functions. Or is this the way to go? I cannot image that .. – Colobus

You did something wrong that does not match the documentation. First off, do not modify the Agent service account - undo any changes. Second, you need to assign the correct role to the service account and to the Functions service itself. Edit your question with details on those two items. Don't say what you tried, show what is exactly configured at this time. – Intestine

@JohnHanley I added the requested details – Colobus

Please show what is set, not what you tried to do. Use the gcloud commands to read the assignments. Show the commands and results in your question. – Intestine

@JohnHanley: added – Colobus

Just for testing if the issue is in fact a Permission's Error or not some error on the configuratoin, have you tried giving the Cloud Scheduler service account the Owner role? – Arsenious

Delete the role roles/cloudscheduler.serviceAgent assigned to the service account (scheduler@...). No permissions are required. The service account is only used in your example for identity. Authorization is determined by Cloud Run on receipt of the OIDC token. What permissions (roles) are assigned to the Cloud Scheduler Service Agent? – Intestine

@DanielOcando: tried it, but didn't work – Colobus

@JohnHanley: I deleted the role, but nothing has changed. Regarding your question: The Cloud Scheduler Service Agent is the only service account being used in this example. I tried it with "no role", "owener role" and "serviceAgent" role, but nothing helped. – Colobus

Have you tried adding the Service Account Token Creator role to App Engine's Default Service Account. Which is the Cloud Function's default runtime account. In your particular case it should have the following name: [email protected] – Arsenious

Didn't work either .. this makes absolutely no sense to me. Do I really have to contact and pay an Google Cloud Consultant, to make this simple thing work? :-) – Colobus

Answer 1

Here are the steps I followed to make Cloud Scheduler trigger an HTTP triggered Cloud Function that doesn't allow unauthenticated invocations:

1. Create a service account, which will have the following form [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com.
2. Adde the service account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com as a project member and added the following roles to the service account: Cloud Functions Invoker and Cloud Scheduler Admin.
3. Deploy an HTTP triggered Cloud Function that doesn't allow public (unauthenticated) access (if you are using the UI, simply uncheck the Allow unauthenticated Invocations checkbox) and that used the recently created service account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com on the Service account field (click more and look for the Service account field, by default it should be set to the App Engine default service account) and take notice of the Cloud Function's URL.
4. Create a Cloud Scheduler job with authentication by issuing the following command from the Cloud Shell: gcloud scheduler jobs create http [JOB-NAME] --schedule="* * * * *" --uri=[CLOUD-FUNCTIONS-URL] --oidc-service-account-email=[SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com

In your specific case you are leaving the default App Engine service account for your Cloud Functions. Change it to the service account you created as specified on the previous steps.

Arsenious

I also tried all that, without success. I then created a new project, deployed everything there, and that everything worked as intended (in fact, every single setup, that we discussed throughout this post). I have no clue, why it hasnt worked before. Maybe there are some constraints on a higher level, that prevented the job from accessing the function .. I dont know .. – Colobus

But according to your last edit, the Function Description: section states that App Engine's Default service account was being used serviceAccountEmail: [email protected] as runtime service account on your Cloud Function, and you needed to use the [email protected] service account when creating the Cloud Function as specified on step 3 of my post. Nonetheless I'm really glad you got it working on a new project! – Arsenious

@DanielOcando Solution given by you worked well in my case. I am triggering authenticated cloud function from cloud scheduler. – Indented

@DanielOcando The solution provided by you works for me in scenario "trigger cloud function from cloud scheduler". I have another situation where I want to trigger authenticated cloud function from another authenticated cloud function. For more information you can refer : #61056472 – Indented

Answer 2

@Marko I went through the same issue, it seems to re-enable (disable/enable) the scheduler API did the fix. This is why creating a new project makes sense because you probably got a scheduler service account by doing so. So if your project doesn't have a scheduler service account created from google, doing this trick will give you one. And although you don't need to assign this specific service account to any of your tasks, it must be available. You can see my work here: How to invoke Cloud Function from Cloud Scheduler with Authentication

Materiality

I still had an explicit service account I created, but disabling and re-enabling api then re-creating my schedule (with my explicit service account) FINALLY fixed the problem. Thanks for the tip. – Repartee

glad it helped! @Matt Byrne – Materiality

Wasted tons of time trying to find missing permissions/roles for my service accounts. Can't believe the "turning it off and on" was the solution after all. – Watercourse

you made it through! nice job, was a frustrating bug for me at the time. – Materiality

@GurupadMamadapur Nowadays we know a culprit: cloud.google.com/scheduler/docs/… See an answer of Vitalii Kyrychenko. – Larocca

Answer 3

I had a similar issue.

In our case, we've enabled Cloud Scheduler quite a long time ago.

According to the docs, if you enabled Cloud Scheduler API before March 19, 2019, you need to manually add the Cloud Scheduler Service Agent role to your Cloud Scheduler service account.

So we had to create a new service account that looks like this service-[project-number]@gcp-sa-cloudscheduler.iam.gserviceaccount.com

Hope this will help anybody else.

Recto

Looks like this works also for Pub/Sub trigger while the documentation is for HTTP – Pompom

I also had to add the "Cloud Scheduler Service Agent" role to the service account that Scheduler calls my function with. – Wormeaten

Answer 4

Question is probably about first generation of Cloud Functions but maybe my struggle will help someone who is trying with second generation CF.

After quite a struggle with this I managed to setup proper permissions to second generation cloud function.

Few things I did not realize from the start:

• when creating second generation function, GCP is also creating Cloud Run service connected to this function
• all http trigger requests comes through Cloud Run service
• Cloud Run SA is the actual caller of Cloud Function (at least this seems to be the case when I tested)
• name of Cloud Run service is generated based on Cloud Function name and in most cases is the same (not when using underscore in CF name, it will replace it with dash my_cf_function => my-cf-function)

So what I did to make it work:

#!/bin/bash

set -e

SA_NAME="my-cf-sa"
PROJECT="my-project-id"
REGION="europe-west1"
SA_EMAIL="${SA_NAME}@${PROJECT}.iam.gserviceaccount.com"
IAM_MEMBER="serviceAccount:${SA_EMAIL}"
CLOUD_FUNCTION_NAME="my-http-function"
SCHEDULER_JOB_NAME="my-cf-job"

# Create SA to use with function

gcloud iam service-accounts create $SA_NAME \
--description="My CF SA" \
--display-name="My CF SA" \
--project=$PROJECT


# Deploy function with created SA

gcloud functions deploy $CLOUD_FUNCTION_NAME \
--gen2 \
--runtime=python311 \
--source=. \
--entry-point=my_http_function \
--trigger-http \
--service-account=$SA_EMAIL \
--run-service-account=$SA_EMAIL \
--no-allow-unauthenticated \
--region=$REGION \
--project=$PROJECT

# COMMAND BELOW DOES NOT WORK even though it is in documentation here https://cloud.google.com/scheduler/docs/http-target-auth
#
# Generate this error:
#
# ERROR: (gcloud.functions.add-iam-policy-binding) ResponseError: status=[400], code=[Ok], message=[Invalid argument: 'An invalid argument was specified. Please check the fields and try again.']
#
# Bug report for this: https://issuetracker.google.com/issues/284853816
#
# gcloud functions add-iam-policy-binding $CLOUD_FUNCTION_NAME --member=$IAM_MEMBER --role=roles/run.invoker --gen2 --region=$REGION --project=$PROJECT


# Add roles/cloudfunctions.invoker to allow SA to invoke function run
#
# After running this command it will ask you if you want to also add iam for Cloud Run service connected to CF.
#
#
#     WARNING: The role [roles/cloudfunctions.invoker] was successfully bound to member [serviceAccount:[email protected]]
#     but this does not grant the member permission to invoke 2nd gen function [my-http-function]. Instead, the role [roles/run.invoker] must be
#     granted on the underlying Cloud Run service. This can be done by running the `gcloud functions add-invoker-policy-binding` command.
#
#     Would you like to run this command and additionally grant [serviceAccount:[email protected]] permission to invoke function [my-http-function] (Y/n)?
#
# If you choose YES you don't need to manually run iam binding command for Cloud Run service.
# I choose NO and did this in two steps so I could replicate all this in terraform later on

gcloud functions add-iam-policy-binding $CLOUD_FUNCTION_NAME \
--member=$IAM_MEMBER \
--role=roles/cloudfunctions.invoker \
--gen2 \
--region=$REGION \
--project=$PROJECT

# Add roles/run.invoker to allow SA invoke Cloud Run service that is connected to function
#
# Only needed if you choose NO in previous command

gcloud run services add-iam-policy-binding $CLOUD_FUNCTION_NAME \
--member=$IAM_MEMBER \
--role=roles/run.invoker \
--region=$REGION \
--project=$PROJECT

# Create Scheduler job with OIDC auth set to use SA

gcloud scheduler jobs create http $SCHEDULER_JOB_NAME \
--schedule="0 */6 * * *" \
--uri="$(gcloud functions describe $CLOUD_FUNCTION_NAME --gen2 --project=$PROJECT --region=$REGION --format="value(serviceConfig.uri)")" \
--http-method=GET \
--oidc-service-account-email=$SA_EMAIL \
--location=$REGION \
--project=$PROJECT

You need to wait few mintes before actually running a job because IAM permissions may take some time to propagate everywhere.

After this I run job manually:

gcloud scheduler jobs run $SCHEDULER_JOB_NAME --location=$REGION --project=$PROJECT

I've checked logs and

gcloud logging read "resource.type=\"cloud_scheduler_job\" AND resource.labels.job_id=\"$SCHEDULER_JOB_NAME\" AND resource.labels.location=\"$REGION\"" --project=$PROJECT --limit 1

And I got 200 from CF

---
httpRequest:
  status: 200
insertId: gde4phfur06nd
jsonPayload:
  '@type': type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished
  jobName: projects/my-project-id/locations/europe-west1/jobs/my-cf-job
  targetType: HTTP
  url: <FUNCTION_URL>
logName: projects/my-project-id/logs/cloudscheduler.googleapis.com%2Fexecutions
receiveTimestamp: '2023-06-03T16:40:28.454858025Z'
resource:
  labels:
    job_id: my-cf-job
    location: europe-west1
    project_id: my-project-id
  type: cloud_scheduler_job
severity: INFO
timestamp: '2023-06-03T16:40:28.454858025Z'

Accuracy

Answer 5

this tutorial helped me to invoke a programmer function, but there is a problem when creating the program after creating the service account, finally eliminating the programmer and doing it again.

Google Cloud Scheduler - Calling Cloud Function

Synesthesia