How to get gsutil to use gcloud credentials in a docker container

Question

Why doesn't gsutil use the Gcloud credentials as it should when running in a docker container on Cloud Shell?

According to [1] gsutil should use gcloud credentials when they are available:

Once credentials have been configured via gcloud auth, those credentials will be used regardless of whether the user has any boto configuration files (which are located at ~/.boto unless a different path is specified in the BOTO_CONFIG environment variable). However, gsutil will still look for credentials in the boto config file if a type of non-GCS credential is needed that's not stored in the gcloud credential store (e.g., an HMAC credential for an S3 account).

This seems to work fine in gcloud installs but not in docker images. The process I used in Cloud Shell is:

docker run -ti --name gcloud-config google/cloud-sdk gcloud auth login
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gcloud compute instances list --project my_project
... (works ok)

docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil ls gs://bucket/

ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket.

[1] https://cloud.google.com/storage/docs/gsutil/addlhelp/CredentialTypesSupportingVariousUseCases

Dolores

Answer 1

You need to mount a volume with your credentials :

docker run -v ~/.config/gcloud:/root/.config/gcloud your_docker_image

Stratocumulus

The credentials are already present via the '--volumes-from' option. – Dolores

my mistake it work on my local machine but not on Cloud Shell – Stratocumulus

When you run docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil version -l there is no config path – Stratocumulus

Answer 2

The following steps solve this problem for me:

1. Set the gs_service_key_file in the [Credentials] section of the boto config file (see here)
2. Activate your service account with gcloud auth activate-service-account
3. Set your default project in gcloud config

Dockerfile snipped:

ENV GOOGLE_APPLICATION_CREDENTIALS=/.gcp/your_service_account_key.json
ENV GOOGLE_PROJECT_ID=your-project-id

RUN echo '[Credentials]\ngs_service_key_file = /.gcp/your_service_account_key.json' \
    > /etc/boto.cfg
RUN mkdir /.gcp
COPY your_service_account_key.json $GOOGLE_APPLICATION_CREDENTIALS


RUN gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS --project $GOOGLE_PROJECT_ID
RUN gcloud config set project $GOOGLE_PROJECT_ID

Anemology

Answer 3

I found @Alexandre's answer basically worked for me, except for one problem: my credentials worked for bq, but not for gsutil (the subject of OP's question), which returned

ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket

How could the same credentials work for one but not the other!?

Eventually I tracked it down: ~/.config/configurations/config_default looks like this:

[core]
account = [email protected]
project = xxxxxxxx
pass_credentials_to_gsutil = false

Why?! Why isn't this documented??

Anyway...change the flag to true, and you're all sorted.

Ligetti