Can you pass a keyfile.json to gsutil?

Question

I have a (maybe unique?) use case in some Python scripts that I am running. Namely, I want the parallel awesomeness of gsutil and so I don't do from google.cloud import storage, rather I use subprocess calls such as:

subprocess.Popen(["gsutil", "-q", "-m", "-o", "GSUtil:parallel_process_count=8,GSUtil:parallel_thread_count=8", "cp", files, destination])

in order to upload and download files from buckets.

In an instance group template I can pass in the service account via -scopes, but I'd like authentication to be handled at the application level. I tried setting environment variables and passing it to subprocess:

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "keyfile.json"
tmp_env = os.environ.copy()
subprocess.Popen(['gsutil', ...], env=tmp_env)

but to no avail. Running:

gcloud auth activate-service-account --key-file /path/to/keyfile.json --project my-project -q

seems to be the best way to authenticate with a json keyfile that does not require the Python API. But it doesn't work if I throw it in at the end of my Dockerfile, and while I could of course throw it in at the end of a startup.sh script that I have executed at the end of an instance group template embedded bootstrap.sh script, neither is really accomplishing what I'd like. Namely, both get away from my original goal of having "gsutil authentication" at the application level.

tl;dr Is there a way to pass keyfile.json credentials to gsutil? Is this a feature the gsutil team has ever discussed? My apologies if I just haven't been hunting the Cloud Platform and gsutil docs well enough.

Duo

Answer 1

You can provide a pointer to a JSON key file for gsutil in your .boto configuration file like so:

[Credentials]
gs_service_key_file = /path/to/your/keyfile.json

This is equivalent to running gsutil config -e for a standalone (non-gcloud) install.

If you want to provide this on the command line as opposed to in your .boto configuration file, you can use the -o parameter similar to how you configured the process and thread counts in your command line. To wit:

subprocess.Popen(["gsutil", "-q", "-m", "-o", "Credentials:gs_service_key_file=/path/to/your/keyfile.json",
                 "-o", "GSUtil:parallel_process_count=8", "-o", GSUtil:parallel_thread_count=8", "cp", files, destination])

Note that you need to make sure the key file path is accessible from within your container.

Cinchonine

Excellent. Exactly what I was looking for. Danke. – Duo

Why on earth can I not find this in documentation? It makes no sense to me that all the documentation for gsutil shows interactive authentication, when I (and surely countless others) need to authenticate from a script on a container somewhere – Limonite

As for why you can't find it in the documentation: that would make sense, since it wasn't mentioned anywhere :( I just pushed a commit to fix that, though! – Arne

This is really good option. Though strangely I could still not find this info anywhere in documentation, and it was a surprise find ! – Silvanasilvano