Google Cloud Tasks cannot authenticate to Cloud Run

Question

I am trying to invoke a Cloud Run service using Cloud Tasks as described in the docs here.

I have a running Cloud Run service. If I make the service publicly accessible, it behaves as expected.

I have created a cloud queue and I schedule the cloud task with a local script. This one is using my own account. The script looks like this

from google.cloud import tasks_v2

client = tasks_v2.CloudTasksClient()

project = 'my-project'
queue = 'my-queue'
location = 'europe-west1'
url = 'https://url_to_my_service'

parent = client.queue_path(project, location, queue)

task = {
        'http_request': {
            'http_method': 'GET',
            'url': url,
            'oidc_token': {
               'service_account_email': '[email protected]'
            }
        }
}

response = client.create_task(parent, task)
print('Created task {}'.format(response.name))

I see the task appear in the queue, but it fails and retries immediately. The reason for this (by checking the logs) is that the Cloud Run service returns a 401 response.

My own user has the roles "Service Account Token Creator" and "Service Account User". It doesn't have the "Cloud Tasks Enqueuer" explicitly, but since I am able to create the task in the queue, I guess I have inherited the required permissions. The service account "[email protected]" (which I use in the task to get the OIDC token) has - amongst others - the following roles:

• Cloud Tasks Enqueuer (Although I don't think it needs this one as I'm creating the task with my own account)
• Cloud Tasks Task Runner
• Cloud Tasks Viewer
• Service Account Token Creator (I'm not sure whether this should be added to my own account - the one who schedules the task - or to the service account that should perform the call to Cloud Run)
• Service Account User (same here)
• Cloud Run Invoker

So I did a dirty trick: I created a key file for the service account, downloaded it locally and impersonated locally by adding an account to my gcloud config with the key file. Next, I run

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://url_to_my_service

That works! (By the way, it also works when I switch back to my own account)

Final tests: if I remove the oidc_token from the task when creating the task, I get a 403 response from Cloud Run! Not a 401... If I remove the "Cloud Run Invoker" role from the service account and try again locally with curl, I also get a 403 instead of a 401.

If I finally make the Cloud Run service publicly accessible, everything works.

So, it seems that the Cloud Task fails to generate a token for the service account to authenticate properly at the Cloud Run service.

What am I missing?

Francois

Me too.. followed docs to the letter: cloud.google.com/tasks/docs/creating-http-target-tasks but am getting 401 responses from the target service. The service account enqueuing the task should only need the permissions 1. Cloud Tasks Enqueuer 2. Service Account User 3. Cloud Run Invoker (Or invoker for whichever google service you're targeting). The enqueueing service account email is added to the task before it is enqueued so that the Cloud Tasks Queue can use it to generate a token.... I'm going to see if this issue resolves in 24 hours like yours did. This is super frustrating – Ceasar

Maybe also worth noting: I recently also got 401 responses when trying to trigger Cloud Run from Cloud Scheduler. I was also using the OIDC token and it turned out that I set the wrong URL in the audience. For Cloud Tasks, it seems like getting the OIDC token happens behind the scenes but I now have the feeling that there is something going wrong there. – Francois

I figured it out. And frustrating thing for me is I've had to solve this issue before... If you don't explicitly populate the audience field for the oidc_token then the target url from the task is used, in your example above: https://url_to_my_service. The problem here is that if you're using Cloud Run with custom domains (instead of the cloud run generated domain), then you'll get an error because OIDC audience doesn't support custom domains. My fix was to explicitly populate the audience with the Cloud Run generated URL, then it worked. – Ceasar

Thank you @Ceasar , this solve the issue for me! I didn't find any documentation about it – Grandiloquence

@Ceasar should I also pass the cloud run generated domain in the main service URL apart from audience? – Grover

@NoopurPhalak see my answer below. My experience was that "I did not need to change the url of the target resource itself" – Ceasar

Answer 1

The next day I am no longer able to reproduce this issue. I can reproduce the 403 responses by removing the Cloud Run Invoker role, but I no longer get 401 responses with exactly the same code as yesterday. I guess this was a temporary issue on Google's side?

Also, I noticed that it takes some time before updated policies are actually in place (1 to 2 minutes).

Francois

Something I just discovered, which is buried in the docs, is that whichever service account you are using to enqueue tasks must have BOTH 'roles/cloudtasks.enqueuer' and 'roles/iam.serviceAccountUser'. The latter which grants permission to tell the Cloud Tasks service to actAs a service account and generate a token. Very convoluted setup compared to PubSub in my opinion. (see step #8)[cloud.google.com/tasks/docs/creating-http-target-tasks#sa] – Redistrict

@Redistrict you are a savior. This solved my problem. – Ealasaid

Answer 2

I had the same issue here was my fix:

Diagnosis: Generating OIDC tokens currently does not support custom domains in the audience parameter. I was using a custom domain for my cloud run service (https://my-service.my-domain.com) instead of the cloud run generated url (found in the cloud run service dashboard) that looks like this: https://XXXXXX.run.app

Masking behavior: In the task being enqueued to Cloud Tasks, If the audience field for the oidc_token is not explicitly set then the target url from the task is used to set the audience in the request for the OIDC token.

In my case this meant that enqueueing a task to be sent to the target https://my-service.my-domain.com/resource the audience for the generating the OIDC token was set to my custom domain https://my-service.my-domain.com/resource. Since custom domains are not supported when generating OIDC tokens, I was receiving 401 not authorized responses from the target service.

My fix: Explicitly populate the audience with the Cloud Run generated URL, so that a valid token is issued. In my client I was able to globally set the audience for all tasks targeting a given service with the base url: 'audience' : 'https://XXXXXX.run.app'. This generated a valid token. I did not need to change the url of the target resource itself. The resource stayed the same: 'url' : 'https://my-service.my-domain.com/resource'

More Reading: I've run into this problem before when setting up service-to-service authentication: Google Cloud Run Authentication Service-to-Service

Ceasar

Thanks for this breakdown, it helped me realise that my issue was audience related. Specifically, using a cloud function as the handler but with a path suffix, I needed to explicitly set the audience sans path suffix for it to work. – Buggs

How did you set your custom URL? Was it using an external static IP load balancer? That's what I have set up and this doesn't seem to fix it. – Mcardle

The custom-url for the service is created in Cloud Run. I set the audience field for the OIDC token with the cloud-run-generated-url, in the request object. – Ceasar

+1 I was struggling for a few hours getting cloud scheduler to talk with Cloud Run Jobs. Turns out OIDC does not work with run.googleapis.com/apis/run.googleapis.com/v1/namespaces. Only OAuth works ! – Benham

@Ceasar HTTP Target Cloud Tasks can be sent to any server (even outside of Google). Doesn't that contraindicate with not supporting custom domains in "audience"? Cloud Task might send request to a non-google machine, so why "run.app" audience would be magic here? – Defalcate

@AlekKowalczyk My guess is that it has something to do with google's implementation of custom domains. I don't know what goes on under the hood in cloud run but I just checked, and Google's docs still say: "Custom domains are currently not supported for the aud value." – Ceasar

Answer 3

1.I created a private cloud run service using this code:

import os

from flask import Flask
from flask import request


app = Flask(__name__)

@app.route('/index', methods=['GET', 'POST'])
def hello_world():
    target = os.environ.get('TARGET', 'World')
    print(target)
    return str(request.data)

if __name__ == "__main__":
    app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
   

2.I created a service account with --role=roles/run.invoker that I will associate with the cloud task

 gcloud iam service-accounts create SERVICE-ACCOUNT_NAME \
 --display-name "DISPLAYED-SERVICE-ACCOUNT_NAME"  
 gcloud iam service-accounts list

 gcloud run services add-iam-policy-binding SERVICE \
 --member=serviceAccount:[email protected] \ 
 --role=roles/run.invoker 

3.I created a queue

gcloud tasks queues create my-queue

4.I create a test.py

from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime

# Create a client.
client = tasks_v2.CloudTasksClient()

# TODO(developer): Uncomment these lines and replace with your values.
project = 'your-project'
queue = 'your-queue'
location = 'europe-west2' # app engine locations
url = 'https://helloworld/index'
payload = 'Hello from the Cloud Task'

# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)

# Construct the request body.
task = {
        'http_request': {  # Specify the type of request.
            'http_method': 'POST',
            'url': url,  # The full url path that the task will be sent to.
            'oidc_token': {
                'service_account_email': "your-service-account"
            },
             'headers' : {
             'Content-Type': 'application/json',
           }
        }
}

# Convert "seconds from now" into an rfc3339 datetime string.
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=60)

# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(d)

# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
task['name'] = 'projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task'


converted_payload = payload.encode()

# Add the payload to the request.
task['http_request']['body'] = converted_payload


# Use the client to build and send the task.
response = client.create_task(parent, task)

print('Created task {}'.format(response.name))
#return response

5.I run the code in Google Cloud Shell with my user account which has Owner role.

6.The response received has the form:

Created task projects/your-project/locations/app-engine-loacation/queues/your-queue/tasks/your-task

7.Check the logs, success

Schwa

Thanks for sharing that. The only difference I see with what I'm doing is the way you bind the role to the service account. I used the console UI while you use a gcloud command. Would that make a difference? – Francois

Answer 4

For those like me, struggling through documentation and stackoverflow when having continuous UNAUTHORIZED responses on Cloud Tasks HTTP requests:

As was written in thread, you better provide audience for oidcToken you send to CloudTasks. Ensure your requested url exactly equals to your resource.

For instance, if you have Cloud Function named my-awesome-cloud-function and your task request url is https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function/api/v1/hello, you need to ensure, that you set function url itself.

{ 
  serviceAccountEmail: [email protected],
  audience: https://REGION-PROJECT-ID.cloudfunctions.net/my-awesome-cloud-function 
}

Otherwise seems full url is used and leads to an error.

Bridges

I am one of those struggling with this audience-related. I just did as you suggest and I still get the same message: Failed to validate auth token. FirebaseAuthError: Firebase ID token has incorrect "aud" (audience) claim. Expected "XXX" but got "https://europe-west3-XXX.cloudfunctions.net/YYY". Make sure the ID token comes from the same Firebase project as the service account used to authenticate this SDK. See https://firebase.google.com/docs/auth/admin/verify-id-tokens for details on how to retrieve an ID token. Does this ring a bell? It was worth the try :) Thanks! – Bloke

Answer 5

Not directly related to the OP issue but I was struggling to call Cloud Workflows from Cloud Tasks. The issue is that I was using OIDC instead of OAuth. It's mentioned in https://cloud.google.com/tasks/docs/reference/rpc/google.cloud.tasks.v2#google.cloud.tasks.v2.CreateTaskRequest

OAuthToken

If specified, an OAuth token will be generated and attached as an Authorization header in the HTTP request.

This type of authorization should generally only be used when calling Google APIs hosted on *.googleapis.com.

The url for workflows executions is starting with workflows.googleapis.com, that's why OAuth should be used instead of OIDC.

Landlubber