Generating Cloud Storage Signed URL from Google Cloud Function without using explicit key file

Question

I'd like to create a pre-signed upload URL to a storage bucket, and would like to avoid an explicit reference to a json key.

Currently, I'm attempting to do this with the Default App Engine Service Account

I'm attempting to follow along with this answer but am getting this error:

AttributeError: you need a private key to sign credentials.the credentials you are currently using <class 'google.auth.compute_engine.credentials.Credentials'> just contains a token. see https://googleapis.dev/python/google-api-core/latest/auth.html#setting-up-a-service-account for more details.

My Cloud Function code looks like this:

from google.cloud import storage
import datetime
import google.auth

def generate_upload_url(blob_name, additional_metadata: dict = {}):
    credentials, project_id = google.auth.default()
    # Perform a refresh request to get the access token of the current credentials (Else, it's None)
    from google.auth.transport import requests

    r = requests.Request()
    credentials.refresh(r)

    client = storage.Client()
    bucket = client.get_bucket("my_bucket")
    blob = bucket.blob(blob_name)
    
    service_account_email = credentials.service_account_email
    print(f"attempting to create signed url for {service_account_email}")
    url = blob.generate_signed_url(
        version="v4",
        service_account_email=service_account_email,
        access_token=credentials.token,
        # This URL is valid for 120 minutes
        expiration=datetime.timedelta(minutes=120),
        # Allow PUT requests using this URL.
        method="PUT",
        content_type="application/octet-stream",
        
    )
    return url


def get_upload_url(request):
    blob_name = get_param(request, "blob_name")
    url = generate_upload_url(blob_name)
    return url

Beni

Answer 1

When you use version v4 of signed URL, the first line of the method calls ensure_signed_credentialsmethod that check if the current service account can generate a signature in standalone mode (so with a private key). And so, that's break the current behavior.

In the comment of the function, it's clearly describe that a service account JSON file is required

        If you are on Google Compute Engine, you can't generate a signed URL.
        Follow `Issue 922`_ for updates on this. If you'd like to be able to
        generate a signed URL from GCE, you can use a standard service account
        from a JSON file rather than a GCE service account.

So, use v2 version instead.

Exchange

It seems that the use of V2 is discouraged per the docs and the suggestion is to use V4. Does that mean there is no getting around an explicit key file? – Beni

I opened an issue, and submitted a fix. I will keep you posted and updated the answer accordingly. – Exchange

In addition, if you look the comment of the v2 signed url method, the warning is the same, the credential required type also and it works. The documentation isn't up to date, at least not sync with the code. Stay tunned – Exchange

Answer 2

As an alternative, you can also use an impersonated service account for creating short-lived credentials with temporary elevated permissions.

Service account impersonation

Requirements:

• Enable the IAM Service Account Credentials API
• Grant the following additional privileges to the service account that will be used for impersonation.
  • Service Account Token Creator
  • Service Account User

Code:

def get_impersonated_credentials():
    scopes=['https://www.googleapis.com/auth/cloud-platform']
    credentials, project = auth.default(scopes=scopes)
    if credentials.token is None:
        credentials.refresh(requests.Request())
    signing_credentials = impersonated_credentials.Credentials(
        source_credentials=credentials,
        target_principal=credentials.service_account_email,
        target_scopes=scopes,
        lifetime=datetime.timedelta(seconds=3600),
        delegates=[credentials.service_account_email]
    )
    return signing_credentials

Then in your function that generates the signed url...

    ...
    credentials=get_impersonated_credentials()
    url = blob.generate_signed_url(
        version="v4",
        credentials=credentials,
        # This URL is valid for 120 minutes
        expiration=datetime.timedelta(minutes=120),
        # Allow PUT requests using this URL.
        method="PUT",
        content_type="application/octet-stream",
   ...
        

Successfully tested on Cloud Run.

Credits: https://blog.salrashid.dev/articles/2021/gcs_signedurl/

Chromosome