Access environment variables stored in Google Secret Manager from Google Cloud Build

Question

How can I access the variables I define in Google Secret Manager from my Google Cloud Build Pipeline ?

Filth

I wouldn't recommend using Google Secret Manager now, because it's in pre-release state which mean that it will for sure change in the future. As for accessing environment variables stored in Google Secret Manager is concerned, there is still no way to actually retrieve it from Cloud Build. Nonetheless, here you will find how to manage encrypted resources within Cloud Build. I hope it helps. – Preliminaries

I not agree with @ChristopherRodriguezConde. Google Beta product are often very close to the production version. Alpha version are subject to change (or to be cancelled), you can be confident in Beta version, in term of stability and availability. – Supinate

Anyway, can you describe your use case for doing this? Why do you need to have your secret in plain text during your Cloud Build pipeline? – Supinate

@guillaumeblaquiere I have a lot of environment variables and I need a more intuitive way of storing them for the different environments. I'd rather not use KMS since it brings about a lot of complexity for a rather simple task. I just want variables in the build, as I would have them in circleci, travisci, bitbucket-pipelines... e.t.c – Filth

Does this answer your question? How to specify secretEnv to cloudbuild.yaml via gcloud cli args or environment variables – Isolt

Answer 1

You can access to secret from Cloud Build by using the standard Cloud Builder gcloud

But, there is 2 issues:

1. If you want to use the secret value in another Cloud Build step, you have to store your secret in a file, the only way to reuse a previous value from one step to another one
2. The current Cloud Builder gcloud isn't up to date (today, 03 feb 2020). You have to add a gcloud component update for using the correct version. I opened an issue for this.

steps:
    - name: gcr.io/cloud-builders/gcloud
      entrypoint: "bash"
      args:
          - "-c"
          - |
              gcloud components update
              # Store the secret is a temporary file
              gcloud beta secrets versions access --secret=MySecretName latest > my-secret-file.txt
    - name: AnotherCloudBuildStepImage
      entrypoint: "bash"
      args:
          - "-c"
          - |
              # For getting the secret and pass it to a command/script
              ./my-script.sh $(cat my-secret-file.txt)

Think to grant the role Secret Manager Secret Accessor roles/secretmanager.secretAccessor to the Cloud Build default service account <PROJECT_ID>@cloudbuild.gserviceaccount.com

EDIT

You can access to the secret from anywhere, either with the gcloud CLI installed (and initialized with a service account authorized to access secrets) or via API call

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
https://secretmanager.googleapis.com/v1beta1/projects/PROJECT_ID/secrets/MySecretName/versions/latest:access

Note: You recieve the secret in the data field, in base64 encoded format. Don't forget to decode it before using it!

You have to generate an access token on a service account with the correct role granted. Here I use again gcloud, because it's easier. But according with your platform, use the most appropriate method. A python script can also do the job.

---

EDIT 2

A new way to get secrets exists now in Cloud Build. Less boiler plate, safer. Have a look and use this way now.

Supinate

Are there any intentions to have this done directly by Cloud Build or MUST I piece together a script to handle getting the variables? It seems like something Google should have automated. – Filth

Cloud Build because it was your question title, but you can do it from where you want. Either with Gcloud command like in my example, or directly with the API call. Look at my edit – Supinate

While I can’t comment on specifics, we are working on deeper integrations with other GCP products. This will be easier in the future, but for now you can use gcloud or the API – Isolt

Looking forward to that future. – Filth

I'm not quite sure if my issue is the same, but I'm following this guide, and on step 4 we create the secret. However on step 7 when running the migration it seems the secret is not being detected: File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 273, in get_value value = self.ENVIRON[var] File "/usr/local/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'SECRET_KEY' codelabs.developers.google.com/codelabs/cloud-run-wagtail#7 – Thomas

Can you suggest passing the secrets between steps with a different entry point? Example is here github.com/vivekkranjan/docker-dataflowrunner-python3/blob/… – Kimura

I edited my post. A new way exists now. @Isolt – Supinate