Kubernetes Secrets Volumes vs Environment Variables

Question

Is there a recommended way to use Kubernetes Secrets? They can be exposed as environment variables or using a volume mount. Is one more secure than the other?

Depilate

Answer 1

https://www.oreilly.com/library/view/velocity-conference-2017/9781491985335/video316233.html

Kubernetes secrets exposed by environment variables may be able to be enumerated on the host via /proc/. If this is the case it's probably safer to load them via volume mounts.

Pym

You can watch free version of the presentation with the same title in HashiCorp's HashiConf: youtube.com/watch?v=j3QJRdiTr1I – Blondell

Answer 2

According to Kubernetes in Action Second Edition book, in some ways secrets stored in secrets-volumes are safer than in environment-variables.

...injecting secrets into environment variables is no different from injecting config maps. But even if Kubernetes allows you to expose secrets in this way, it may not be the best idea, as it can pose a security risk. Applications typically output environment variables in error reports or even write them to the application log at startup, which can inadvertently expose secrets if you inject them into environment variables. Also, child processes inherit all environment variables from the parent process. So, if your application calls a third-party child process, you don’t know where your secrets end up.

Corked

Answer 3

I agree with TMCs answer, but wanted to add a note for those that are thinking, "But what about 12-factor??". Objections are sometimes raised against using volume-mounted secrets because 12F seemingly requires configs be stored as ENV vars. First, these are suggested, voluntary, your-mileage-may-vary best-practices suggestions. Second, there is this section:

In a twelve-factor app, env vars are granular controls, each fully orthogonal to other env vars. They are never grouped together as “environments”, but instead are independently managed for each deploy. This is a model that scales up smoothly as the app naturally expands into more deploys over its lifetime.

source: https://12factor.net/config

Basically, coupled with the rest of the description I understand the guiding principles of 12F Config management to be:

• Keep config out of source
• Be able to inject config into source artifact (e.g. a docker container)
• Be able to make granular changes to the set of required configuration values

In my humble opinion, volume mounted Kubernetes Secrets can accomplish these goals depending on what sort of Secret objects you create and how you manage them.

Grijalva

Answer 4

Mounted Secrets are updated automatically

• When a secret being already consumed in a volume is updated, projected keys are eventually updated as well. Kubelet is checking whether the mounted secret is fresh on every periodic sync. However, it is using its local cache for getting the current value of the Secret. 

• in an multi container pod, each container inside a pod has to request the secret volume in its volumeMounts for it to be visible within the container. This can be used to construct useful security partition at pod level. 

With above finding from official docs secret by volume mount look a better option.

Organization

Answer 5

I don't think there is any difference in safety. Because If a node is compromised, they can see the secret.

example:

---
apiVersion: v1
kind: Secret
metadata:
  name: mount
type: Opaque
data:
  foo: ""

---
apiVersion: v1
kind: Secret
metadata:
  name: env
type: Opaque
data:
  BAR: "BAR"


---
apiVersion: v1
kind: Pod
metadata:
  name: mount
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
      - containerPort: 80
      envFrom:
        - secretRef:
            name: env
      volumeMounts:
      - name: mount
        mountPath: "/opt/mount"
        readOnly: true
  volumes:
  - name: mount
    secret:
      secretName: mount

$ kubectl exec mount -- ls -la /opt/mount
total 0
drwxrwxrwt 3 root root 100 Jan  8 03:00 .
drwxr-xr-x 1 root root  19 Jan  8 03:00 ..
drwxr-xr-x 2 root root  60 Jan  8 03:00 ..2020_01_08_03_00_13.066710719
lrwxrwxrwx 1 root root  31 Jan  8 03:00 ..data -> ..2020_01_08_03_00_13.066710719
lrwxrwxrwx 1 root root  10 Jan  8 03:00 foo -> ..data/foo

$ kubectl exec mount -- env | grep BAR
BAR=

$ docker ps | grep mount
8dbde26864a4        nginx                                                                                "nginx -g 'daemon of…"   8 minutes ago       Up 8 minutes                            k8s_nginx_mount_default_3438a94b-e4af-41a7-8d85-7668fcbd9928_0
$ docker inspect 8dbde26864a4 | grep -A 1 '"Env"'
            "Env": [
                "BAR=",
$ docker inspect 8dbde26864a4 | grep '"Source"' | grep mount
"Source": "/var/lib/kubelet/pods/3438a94b-e4af-41a7-8d85-7668fcbd9928/volumes/kubernetes.io~secret/mount"
$ ls -la /var/lib/kubelet/pods/3438a94b-e4af-41a7-8d85-7668fcbd9928/volumes/kubernetes.io~secret/mount
合計 0
drwxrwxrwt 3 root root 100  1月  8 12:00 .
drwxr-xr-x 4 root root  46  1月  8 12:00 ..
drwxr-xr-x 2 root root  60  1月  8 12:00 ..2020_01_08_03_00_13.066710719
lrwxrwxrwx 1 root root  31  1月  8 12:00 ..data -> ..2020_01_08_03_00_13.066710719
lrwxrwxrwx 1 root root  10  1月  8 12:00 foo -> ..data/foo

And you can see the secret design proposal https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/secrets.md

It is written like this

If a node is compromised, the only secrets that could potentially be exposed should be the secrets belonging to containers scheduled onto it

So I think the secret does not seem to guarantee the protection of the container's secret when a node is compromised.

Moores

Answer 6

I've also had the same question and have been looking for a definitive answer regarding environment variables vs. volumes. I can't find anything, just discussions like these. The Kuberentes documentation doesn't address this either, except to noted by DT. above, and even that is lacking a bit. Addressing some of what's discussed here I have my opinions, which aren't necessarily correct, just what I've been able to surmise. If I'm misunderstanding something I welcome corrections. Here's my understanding:

1. Like bells17 I don't see any difference in security between environment vars and volumes/files, both are easily accessible just by logging into the pod.
2. WRT 12 factor apps, I agree with jamesconant. I don't think there is any difference. In both cases they're part of the secret specification which is the source of the information regardless of whether it gets projected as a file or an environment variable. You can put the secret spec (and other specs) in a separate repo if needed to fully separate code from configuration.
3. Secrets are not secure unless encrypted at rest in etcd. As noted by Andre B other options like Vault are possible.

Borneo