Application Default Credentials in Google Cloud Build

Question

Within my code, I am attempting to gather the Application Default Credentials from the associated service account in Cloud Build:

from google.auth import default

credentials, project_id = default()

This works fine in my local space because I have set the environment variable GOOGLE_APPLICATION_CREDENTIALS appropriately. However, when this line is executed (via a test step in my build configuration) within Cloud Build, the following error is raised:

google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. 
Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. 
For more information, please see https://cloud.google.com/docs/authentication/getting-started

This is confusing me because, according to the docs:

By default, Cloud Build uses a special service account to execute builds on your behalf. This service account is called the Cloud Build service account and it is created automatically when you enable the Cloud Build API in a Google Cloud project. Read Here

If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set, ADC uses the service account that is attached to the resource that is running your code. Read Here

So why is the default call not able to access the Cloud Build service account credentials?

Coccidioidomycosis

What's your dependency version? Can you show the step where you call your python script? – Givens

Answer 1

There is a trick: you have to define the network to use in your Docker build. Use the parameter --network=cloudbuild, like that

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: 'docker'
    args: 
      - build
      - '--no-cache'
      - '--network=cloudbuild'
      - '-t'
      - '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
      - .
      - '-f' 
      - 'Dockerfile'
...

You can find the documentation here

Givens

This is also a good answer and in some situations the better answer. – Alyss

This is a life saver! I doubted it at first, but it's magic. This should be accepted answer. – Savage

We are using kaniko and they don't even include this flag, so this sucks. Google really did not think things through with workload identity when it comes to CI/CD. – Irvin

Same for buildx – Tridentum

For buildx, we've been using this image successfully for some time: github.com/MisoRobotics/cloudbuildx – Packaging

This worked for me to build a nextjs app that uses firebase-admin to manage user accounts. Before was failing at the "Generating static pages" step on the next build command – Beget

This should be the accepted answer. it solves the issue without adding security concerns that go with a service account key. – Incorrect

Answer 2

Application Default Credentials (ADC) defines the method for searching for credentials.

The Cloud Build VM instance is fetching credentials from metadata via network calls to 169.254.169.254. The container you are running does not have access to the host's network, meaning the code running inside the Docker container cannot access the host's metadata. Since there are no other credentials inside your container, ADC fails with the message Could not automatically determine credentials.

Solution: Provide credentials there are accessible inside the Docker container.

One method is to store a service account JSON key file in Cloud Storage. Then configure Cloud Build to download the file. Configure the Dockerfile to copy the file into the image.

Cloud Build YAML:

- name: gcr.io/cloud-builders/gsutil
  args: ['cp', 'gs://bucketname/path/service-account.json', 'service-account.json']

Dockerfile:

COPY ./service-account.json /path/service-account.json

Google provides additional services such as Secret Manager that can also be used. I prefer Cloud Storage as the ease of storing and updating credentials provides for easy documentation and management.

In considering issues regarding security, separation of privilege, and management Google Cloud offers several methods.

Another method is the one posted by Guillaume Blaquiere.

In environments with relaxed security and developers are granted broad permissions (IAM Roles), Guillaume's answer is simple and easy to implement. In tightly controlled security environments, granting Cloud Build broad permissions is a security risk.

Security often is a tradeoff between security, implementation, and ease of use. Granting IAM Roles requires careful thought on how permissions are to be used and by what/who.

Alyss

Hey my friend, I always disagree when a solution uses a service account key file, it makes me sick!! I already discussed with the Cloud Build team about that point and they provided me the solution (provided in my answer) already documented and weirdly known. Anyway, it works great! – Givens

@guillaumeblaquiere - Since the container is built by Cloud Build, using a service account JSON key file is secure - maybe even more secure because Cloud Build requires fewer permissions in its service account. However, you are correct, service account key files require knowledge and management to be implemented correctly. There is often a tradeoff between security, feasibility, and convenience. Too bad we cannot combine answers. – Alyss

No it isn't secure, it stores it in the docker image layer. That's not what we want. The idea is for cloudbuild to automatically act as the GOOGLE_APPLICATION_CREDENTIALS within any operation that is scripted in the Dockerfile, such as a bundle install for ruby, where secrets manager is initialized. If Google can't address CI/CD then the entire point of using workload identity is defeated in a single step. Unfortunately we use kaniko which doesn't include the Docker flag. – Irvin

@NathanMcKaskle - As my answer explains there are several methods. The key is that some APIs require the private key which might not be available from default credentials. Regarding credentials in a Docker layer that is easy to solve via a multi-stage build. The key is to know the available solutions and then chose one that meets your objectives. There is no one solution that fits all use cases. – Alyss

Answer 3

Cloud Build can securing builds by encrypting your key and decrypting when build using Cloud KMS

After setup encrypted key, you can retrieve it by a build step in cloudbuild.yaml

steps:
- name: gcr.io/cloud-builders/gcloud
  args:
  - kms
  - decrypt
  - "--ciphertext-file=ENCRYPTED_PASSWORD_FILE"
  - "--plaintext-file=PLAINTEXT_PASSWORD_FILE"
  - "--location=global"
  - "--keyring=KEYRING_NAME"
  - "--key=KEY_NAME"
- name: gcr.io/cloud-builders/docker
  entrypoint: bash
  args:
  - "-c"
  - docker login --username=DOCKER_USERNAME --password-stdin < PLAINTEXT_PASSWORD_FILE

Bukavu