How to inject secret from Google Secret Manager into Kubernetes Pod as environment variable with Spring Boot?

Question

For the life of Bryan, how do I do this?

Terraform is used to create an SQL Server instance in GCP. Root password and user passwords are randomly generated, then put into the Google Secret Manager. The DB's IP is exposed via private DNS zone.

How can I now get the username and password to access the DB into my K8s cluster? Running a Spring Boot app here.

This was one option I thought of:

In my deployment I add an initContainer:

- name: secrets
  image: gcr.io/google.com/cloudsdktool/cloud-sdk
  args: 
  - echo "DB_PASSWORD=$(gcloud secrets versions access latest --secret=\"$NAME_OF_SECRET\")" >> super_secret.env

Okay, what now? How do I get it into my application container from here?

There are also options like bitnami/sealed-secrets, which I don't like since the setup is using Terraform already and saving the secrets in GCP. When using sealed-secrets I could skip using the secrets manager. Same with Vault IMO.

Libel

You'd need to use a tool like Berglas: github.com/GoogleCloudPlatform/berglas – Paphos

Do you generate the password and put it into secret manager in Terraform? – Welford

Answer 1

You can use spring-cloud-gcp-starter-secretmanager to load secrets from Spring application itself.

Documentation - https://cloud.spring.io/spring-cloud-gcp/reference/html/#secret-manager

Callimachus

Have you tried this? Does this work with Workload Identities? Thanks! – Salyers

I use it with service accounts, not workload identities. I have terraform code, that creates service account, key for it and stores key as secret in k8s, then application uses that secret to configure any of spring-cloud-gcp projects – Callimachus

Answer 2

On top of the other answers and suggestion in the comments I would like to suggest two tools that you might find interesting.

First one is secret-init:

secrets-init is a minimalistic init system designed to run as PID 1 inside container environments and it`s integrated with multiple secrets manager services, e.x. Google Secret Manager

Second one is kube-secrets-init:

The kube-secrets-init is a Kubernetes mutating admission webhook, that mutates any K8s Pod that is using specially prefixed environment variables, directly or from Kubernetes as Secret or ConfigMap.

It`s also support integration with Google Secret Manager:

User can put Google secret name (prefixed with gcp:secretmanager:) as environment variable value. The secrets-init will resolve any environment value, using specified name, to referenced secret value.

Here`s a good article about how it works.

Chanty

Neither of these 2 systems worked for me in my app. I tried them both. They both errored out and there were no errors specified in the logs. – Breda

I ended up using github.com/external-secrets/kubernetes-external-secrets. – Salyers

Answer 3

How do I get it into my application container from here?

You could use a volume to store the secret and mount the same volume in both init container and main container to share the secret with the main container from the init container.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: my-app:latest
    volumeMounts:
    - name: config-data
      mountPath: /data
  initContainers:
  - name: secrets
    image: gcr.io/google.com/cloudsdktool/cloud-sdk
    args: 
    - echo "DB_PASSWORD=$(gcloud secrets versions access latest --secret=\"$NAME_OF_SECRET\")" >> super_secret.env
    volumeMounts:
    - name: config-data
      mountPath: /data
  volumes:
  - name: config-data
    emptyDir: {}

Southerner

How do I read it back into the Spring Boot app though? – Salyers

I have not tried this in practice but you can read it from File system using spring.cloud.kubernetes.config.paths or implement a PropertySource – Southerner

Answer 4

Using volumes of emptyDir with medium: Memory to guarantee that the secret will not be persisted.

...
volumes:
      - name: scratch
        emptyDir:
          medium: Memory
          sizeLimit: "1Gi"
...

Cooksey

How to read it back into env vars on Spring Boot side though? – Salyers

Inspired by https://mcmap.net/q/45617/-set-environment-variables-from-file-of-key-value-pairs, you have to read the file before you start spring-boot. So in cmd should have this line before java -jar xxx.jar: export $(grep -v '^#' /blah/blah/super-secrets.env | xargs -d '\n') then the rest read the variable as how spring-boot do. – Cooksey

Answer 5

If one has control over the image, it's possible to change the entry point and use berglas.

Dockerfile:

FROM adoptopenjdk/openjdk8:jdk8u242-b08-ubuntu  # or whatever you need

# Install berglas, see https://github.com/GoogleCloudPlatform/berglas
RUN mkdir -p /usr/local/bin/
ADD https://storage.googleapis.com/berglas/main/linux_amd64/berglas /usr/local/bin/berglas
RUN chmod +x /usr/local/bin/berglas

ENTRYPOINT ["/usr/local/bin/berglas", "exec", "--"]

Now we build the container and test it:

docker build -t image-with-berglas-and-your-app .
docker run \
    -v /host/path/to/credentials_dir:/root/credentials \
    --env GOOGLE_APPLICATION_CREDENTIALS=/root/credentials/your-service-account-that-can-access-the-secret.json \
    --env SECRET_TO_RESOLVE=sm://your-google-project/your-secret  \
    -ti image-with-berglas-and-your-app env

This should print the environment variables with the sm:// substituted by the actual secret value.

In K8s we run it with Workload Identity, so the K8s service account on behalf of which the pod is scheduled needs to be bound to a Google service account that has the right to access the secret.

In the end your pod description would be something like this:

apiVersion: v1
kind: Pod
metadata:
  name: your-app
spec:
  containers:
  - name: your-app
    image: image-with-berglas-and-your-app
    command: [start-sql-server]
    env:
      - name: AXIOMA_PASSWORD
        value: sm://your-google-project/your-secret

Welford