how do you manage secret values with docker-compose v3.1?

Question

Version 3.1 of the docker-compose.yml specification introduces support for secrets.

I tried this:

version: '3.1'

services:
  a: 
    image: tutum/hello-world
  secret: 
    password: the_password
  b:
    image: tutum/hello-world

$ docker-compose up returns:

Unsupported config option for services.secret: 'password'

How can we use the secrets feature in practice?

Sleeping

Are you sure docker-compose does already support secrets? Which docker-compose version are you running? – Expressive

$ docker-compose --version returns: docker-compose version 1.11.0, build 6de1806, so yes, it should support secrets according to the release notes. – Sleeping

late to the party, but isn't it supposed to be secrets: and not secret: ? – Espalier

Answer 1

You can read the corresponding section from the official documentation.

To use secrets you need to add two things into your docker-compose.yml file. First, a top-level secrets: block that defines all of the secrets. Then, another secrets: block under each service that specifies which secrets the service should receive.

As an example, create the two types of secrets that Docker will understand: external secrets and file secrets.

1. Create an 'external' secret using docker secret create

First thing: to use secrets with Docker, the node you are on must be part of a swarm.

$ docker swarm init

Next, create an 'external' secret:

$ echo "This is an external secret" | docker secret create my_external_secret -

(Make sure to include the final dash, -. It's easy to miss.)

2. Write another secret into a file

$ echo "This is a file secret." > my_file_secret.txt

3. Create a docker-compose.yml file that uses both secrets

Now that both types of secrets are created, here is the docker-compose.yml file that will read both of those and write them to the web service:

version: '3.1'

services:
  web:
    image: nginxdemos/hello
    secrets:                    # secrets block only for 'web' service
     - my_external_secret
     - my_file_secret

secrets:                        # top level secrets block
  my_external_secret:
    external: true
  my_file_secret:
    file: my_file_secret.txt

Docker can read secrets either from its own database (e.g. secrets made with docker secret create) or from a file. The above shows both examples.

4. Deploy your test stack

Deploy the stack using:

$ docker stack deploy --compose-file=docker-compose.yml secret_test

This will create one instance of the web service, named secret_test_web.

5. Verify that the container created by the service has both secrets

Use docker exec -ti [container] /bin/sh to verify that the secrets exist.

(Note: in the below docker exec command, the m2jgac... portion will be different on your machine. Run docker ps to find your container name.)

$ docker exec -ti secret_test_web.1.m2jgacogzsiaqhgq1z0yrwekd /bin/sh

# Now inside secret_test_web; secrets are contained in /run/secrets/
root@secret_test_web:~$ cd /run/secrets/

root@secret_test_web:/run/secrets$ ls
my_external_secret  my_file_secret

root@secret_test_web:/run/secrets$ cat my_external_secret
This is an external secret

root@secret_test_web:/run/secrets$ cat my_file_secret
This is a file secret.

If all is well, the two secrets we created in steps 1 and 2 should be inside the web container that was created when we deployed our stack.

Adman

I get this result when I run the first command you specify: Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. I'm confused! Do I need to start a swarm in order to use docker-compose with secrets? – Sleeping

Oops, yes you do. The docker stack deploy command is part of the Swarm engine. I'll add a line in Step 1 to indicate that. – Adman

I don't understand the reasoning behind that. Mine and many other use cases don't require a swarm, but do require secrets. Why make us use a swarm if we just want secrets? – Sleeping

@Sleeping If you don't require swarm, you could just mount the secret file from your local machine. Why do you need docker secrets? – Winkler

@Winkler because I need secrets to start containers on my remote machine, not just my local machine. for example, the official owncloud image has a docker-compose.yml that asks you to write down the MySQL password as an environment variable, which is bad practice. I thought docker secrets would solve that? – Sleeping

@Sleeping Yes. But secrets still need to be stored somewhere. In case of docker secrets they're stored in a cluster storage. If you don't use a cluster, you can just create a file and mount it. You don't need the docker secrets feature. – Winkler

Hm, documentation makes it look like you should create a secret from a file, just like you create from simple string, but as demonstrated here, the secret file must be available to the stack deploy command. Am I not understanding something about the file option? – Realpolitik

Nm, it just hit me: external secrets (file or just string) is like external networks, and the other option lets the deploy command create it on the fly for the stack, like a default-created overaly network. nice. – Realpolitik

Secrets are usable in docker-compose w/o swarm as of 1.11 if you use file not external, but note they are not secure, because compose isn't a production tool, and like said here, there's no place to store them encrypted without swarm raft db. If you define file-based secrets properly in compose file, docker-compose will bind-mount the file into /run/secrets to emulate what swarm does for easier developer workflow when working locally. If you need a secure storage on single-node server then you can init a single-node swarm and it works fine. – Fulgurant

It's instructive to run this with docker-compose, as you get a couple of warnings, about the external secret. In the container, you can see the file-based secret in /run/secrets, but not the external one. – Sauerbraten

@BretFisher: What do you mean by "compose isn't a production tool"? The docker documentation has a section dedicated to "Use Compose in production" if someone is using a single server. docs.docker.com/compose/production – Insubstantial

In case it's not obvious to anyone else, the docker-compose secrets are not accessible to the docker build. See this – Zeringue

How are docker secrets secure if you can read them from the console when exec into the container? It looks like it's helpful only from committing to version control. – Flowered

@Flowered - they are not accessible to the console, they only get mounted during build and should not be available afterwards. I haven't tried the above sample, so maybe docker stack deploy does things differently ... but I have used secrets in docker compose and I can confirm the secrets work during build (specify the secret in each step that needs it) but not afterwards. – Wallah

@zholinho: They are not stored in image and only available during runtime. – Magnien

@BretFisher: is there a way to specify target when mounting secret? – Magnien

Answer 2

Given you have a service myapp and a secrets file secrets.yml:

Create a compose file:

version: '3.1'

services:
  myapp:
    build: .
    secrets:
      secrets_yaml

Provision a secret using this command:

docker secret create secrets_yaml secrets.yml

Deploy your service using this command:

docker deploy --compose-file docker-compose.yml myappstack

Now your app can access the secret file at /run/secrets/secrets_yaml. You can either hardcode this path in your application or create a symbolic link.

---

The different question

This answer is probably to the question "how do you provision your secrets to your docker swarm cluster".

The original question "how do you manage secret values with docker compose" implies that the docker-compose file contains secret values. It doesn't.

There's a different question: "Where do you store the canonical source of the secrets.yml file". This is up to you. You can store it in your head, print on a sheet of paper, use a password manager, use a dedicated secrets application/database. Heck, you can even use a git repository if it's safely secured itself. Of course, never store it inside the system you're securing with it :)

I would recommend vault. To store a secret:

# create a temporary secret file
cat secrets.yml | vault write secret/myappsecrets -

To retrieve a secret and put it into your docker swarm:

vault read -field=value secret/myappsecrets | docker secret create secrets_yaml -

Of course, you can use docker cluster itself as a single source of truth for you secrets, but if your docker cluster breaks, you'd lost your secrets. So make sure to have a backup elsewhere.

---

The question nobody asked

The third question (that nobody asked) is how to provision secrets to developers' machines. It might be needed when there's an external service which is impossible to mock locally or a large database which is impossible to copy.

Again, docker has nothing to do with it (yet). It doesn't have access control lists which specify which developers have access to which secrets. Nor does it have any authentication mechanism.

The ideal solution appears to be this:

• A developer opens some web application.
• Authenticates using some single sign on mechanism.
• Copies some long list of docker secret create commands and executes them in the terminal.

We have yet to see if such an application pops up.

Winkler

docker secret create seems to require that there be a pre-existing swarm? do I need to create one? – Sleeping

@Sleeping So you're running this as a developer? I'm afraid Docker doesn't have support for that use case yet. But yeah, you could create a docker swarm consisting of only your developer machine. That's out of scope of this question. – Winkler

As of Feb 8th docker-compose 1.11 supports file-based secrets in compose files for local dev. See my comment above on chosen answer for details :) – Fulgurant

@BretFisher but what's the point if it's essentially the same as specifying using volume's ./file_based_secret:/run/secrets/my_secret ? – Winkler

@Winkler Right, there's no functional difference in container. It's about seamless workflow, and limiting the need for multiple compopse files. – Fulgurant

Answer 3

You can also specify secrets stored locally in a file using file: key in secrets object. Then you don't have to docker secret create them yourself, Compose / docker stack deploy will do it for you.

version: '3.1'

secrets:
  password:
    file: ./password

services:
  password_consumer:
    image: alpine
    secrets:
      - password

Reference: Compose file version 3 reference: Secrets

Keaton

How to define multiple variables in the password file? – Tanhya

How do you access the secret in your code/app? I am looking to assign it to a environmental variable if possible. – Floyfloyd

Answer 4

One question was raised here in the comments, why should I initialize a swarm if I only need secrets? And my answer is that secrets is created for the swarm, where you have more than one node and you want to manage and share secrets in a secure way. But if you have one node, this will not (almost) add any extra security if someone can access your host machine where you have the one node swarm, as secrets can be retrieved from the running containers, or directly on the host if the secret is created from a file, like a private key.

Check this blog: https://www.docker.com/blog/docker-secrets-management/

And read the comments: "Thank you very much for the introductory article. The steps are mentioned to view the contents of secrets in container will not work when the redis container is created on a worker node."

Cartridge

Please make more obvious how this answers the question at the top of the page (instead of a question asked in the comments). – Women

Answer 5

For local machine I ended up having 2 .env files: .env, secrets.env (sensitive envs, this file is in .gitignore). Starting from compose version 2.17.0-rc.1 you can specify multiple env files and use it like sudo docker compose --env-file .env --env-file secrets.env up

Pros:

1. You can store all secrets in one file instead of creating multiple secrets.
2. You don’t need swarm.

Cons:

1. More characters in command.
2. When you invoke commands like sudo docker compose logs you will need to specify --env-file .env --env-file secrets.env. Otherwise you will see errors like WARN[0000] The "VAR_NAME" variable is not set. Defaulting to a blank string.

Allison

Answer 6

Is that the exact indentation of your docker-compose.yml file? I think secret secrets should be nested under a (i.e. one of the services), not directly under services section.

Oxygen

yes, that was the exact indentation. I tried nesting the secret dictionary under a (and also at the same level as services) and got the same result. – Sleeping

Answer 7

I struggled with how to manage secrets in docker compose, and ended up writing a little service to do it: dcsm. It lets you store secrets in a file encrypted with age symmetric encryption. When your cluster boots, dcsm decrypts the secrets and injects them into *.template files. You can template configs your services need, or files you provide to the service with env_file.

In this approach, the only thing you need to manage out of band is the encryption key -- everything else is stored in the repo alongside your service configs, the compose.yml file, and the secrets themselves.

Wanwand

Answer 8

I guess the keyword is secrets not secret. That is at least what I understand from reading the schema.

Expressive

the keyword is secret only docs.docker.com/engine/reference/commandline/secret_inspect – Leilani

@Expressive is right about docker-compose: it uses the plural "secrets" Partha is right about docker: it uses singular "secret" – Wite

Answer 9

The keyword is secrets instead of secret. It should also properly indented under service a.

Arquebus

This looks more like a comment to another answer rather than a new answer. – Chrisoula