Blob.generate_signed_url() failing to AttributeError

Question

So I'm trying to produce temporary globally readable URLs for my Google Cloud Storage objects using the google-cloud-storage Python library (https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html) - more specifically the Blob.generate_signed_url() method. I doing this from within a Compute Engine instance in a command line Python script. And I keep getting the following error:

AttributeError: you need a private key to sign credentials.the credentials you are currently using <class 'oauth2cl
ient.service_account.ServiceAccountCredentials'> just contains a token. see https://google-cloud-python.readthedocs
.io/en/latest/core/auth.html?highlight=authentication#setting-up-a-service-account for more details.

I am aware that there are issues with doing this from within GCE (https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/50) but I have created a new Service Account credentials following the instructions here: https://cloud.google.com/storage/docs/access-control/create-signed-urls-program and my key.json file most certainly includes a private key. Still I am seeing that error.

This is my code:

keyfile = "/path/to/my/key.json"
credentials = ServiceAccountCredentials.from_json_keyfile_name(keyfile)
expiration = timedelta(3) # valid for 3 days
url = blob.generate_signed_url(expiration, method="GET",
                               credentials=credentials) 

I've read through the issue tracker here https://github.com/GoogleCloudPlatform/google-cloud-python/issues?page=2&q=is%3Aissue+is%3Aopen and nothing related jumps out so I am assuming this should work. Cannot see what's going wrong here.

Fletcherfletcherism

I am currently hunting the solution for this, as well, because I need a way to sign upload URLs without physical access to a key file. I got this working when a service account key (.json) is configured to GOOGLE_APPLICATION_CREDENTIALS, but, perhaps like you, I want my code to operate with the default credentials implicit to instances on GCP. Did you solve it? – Coulomb

When connecting local machine to storage, we can use credentials from json. But when reading file on app engine from storage- i dont know how to get credentials. docs say default service account should just work fine. But I always get error ' need private key to sign' – Inauspicious

Answer 1

I was having the same issue. Ended up fixing it by starting the storage client directly from the service account json.

storage_client = storage.Client.from_service_account_json('path_to_service_account_key.json')

I know I'm late to the party but hopefully this helps!

Bunker

You saved my life! For anyone else reading this in the future, I found this only applied to creation of v2 signed URLs, when using v4 this did not apply – Resume

Doesn't work; that's what I did to get the error in the first place. – Thibaud

doesn't match with workload identity enabled – Aggregation

What if I dont want to save sv account key in GIT. is there a way ? – Inauspicious

worked in v4 for me – Stanton

This worked in v4 for me too. Also if you already have the service account as a dictionary loaded, you can use from_service_account_info – Heading

Answer 2

Currently, it's not possible to use blob.generate_signed_url without explicitly referencing credentials. (Source: Class Blob - generate_signed_url) However, you can do a workaround, as seen here, which consists of:

signing_credentials = compute_engine.IDTokenCredentials(
    auth_request,
    "",
    service_account_email=credentials.service_account_email
)
signed_url = signed_blob_path.generate_signed_url(
    expires_at_ms,
    credentials=signing_credentials,
    version="v4"
)

Magnesium

Thanks! I had to put the service_account_email in the environment, as credentials.service_account_email was coming back as default (I think) – Collayer

This code works tho afterwards I had to follow this to get the right permissions in google cloud https://mcmap.net/q/420274/-unable-to-assign-iam-serviceaccounts-signblob-permission – Rodenhouse

How do I get this "auth_request" variable? – Marinna

Answer 3

A much complete snippet for those asking where other elements come from. cc @AlbertoVitoriano

    from google.auth.transport import requests
    from google.auth import default, compute_engine
    
    credentials, _ = default()
    
    # then within your abstraction
    auth_request = requests.Request()
    credentials.refresh(auth_request)
    
    signing_credentials = compute_engine.IDTokenCredentials(
        auth_request,
        "",
        service_account_email=credentials.service_account_email
    )
    signed_url = signed_blob_path.generate_signed_url(
        expires_at_ms,
        credentials=signing_credentials,
        version="v4"
    )

Titre

Answer 4

You can run GCloud SDK outside of GCP using Application Default Credentials (as you do with your user account) and make it work as a Service Account by "impersonating the SA". Notice, though, that for this to work you need to have the "Service Account Token Creator" role, which will allow you to create tokens "impersonating" another account (the Service Account).

When you have the permission, run the gcloud auth ADC login procedure, but "upgrading" you ADC credentials to an impersonated SA, like this:

$ gcloud auth application-default login --impersonate-service-account=[your-service-account-id]@developer.gserviceaccount.com

After that, your local ADC credentials are tied to the said SA. Keep in mind that audit trails for operations performed with this SA keep record of who requested the impersonation (ie. you). You can check it by comparing your current ADC credentials file with the newly generated above.

This way, your code will run outside GCP exactly the same than on GCP, without any conditional code and without having to juggle with JSON credentials, which is highly discouraged by Google.

Fluting

Great answer. Adding the corresponding references to the official documentation would be a nice improvement. – Residuum