Docker Swarm with image versions externalized to .env file

Question

I used to externalized my image versions to my .env file. This make it easy to maintain and I don't modify my docker-compose.yml file just to upgrade a version, so I'm sure I won't delete a line by mistake or whatever.

But when I try to deploy my services with stack to the swarm, docker engine complains that my image is not a correct reposity/tag, with the exact following message :

Error response from daemon: rpc error: code = 3 desc = ContainerSpec: "GROUP/IMAGE:" is not a valid repository/tag

To fix this, I can fix the image version directly in the docker-compose.yml file. Is there any logic here or it that a bug? But this mixes fix part of the docker-compose and variable ones.

Cheers, Olivier

Longshore

Answer 1

As already mentioned the .env isn't currently supported by the docker stack. So the alternative way is to clearly specify the environment file using by env_file, e.g.:

version: '3.3'
services:
  foo-service:
    image: foo-image
    env_file:
      - .env
    environment:
      - SOME_ENV=qwerty
      - OTHER_ENV=${SOME_ENV} # Not supported

Update: as per 2024 it still does not support variable substitution neither in env_file nor in environment compose file entry.

Opposite to other answers here it would not either work for docker compose --file your.compose.yml config > processed.yml followed by docker stack deploy --compose-file processed.yml your-stack because the compose file formats for stack and compose mismatch (name, depends_on keys at least).

Prowler

Does it support variable substitution in the .env file, e.g. PGDATABASE=${DATABASE_HOST} when used like this? – Nickeliferous

I just tried it out with a small example and the answer is no :( – Vassalage

@RogerCollins, what is docker version do you have? What is a version of a docker-compose file? How looks like your .env file? Please, provide more details – Prowler

@Prowler docker versions >18.09 - docker stack does not support env_file. See the issue: github.com/moby/moby/issues/29133 – Pulsate

You are confusing .env (used in the docker-compose template itself) and env_file (passed into the containers) – Vainglorious

Answer 2

The yaml parser in docker stack deploy doesn't have all the same features of that in docker-compose. However, you can use docker-compose config to output a yaml file after it's done all the variable substitutions, extending other files, and merging multiple files together. This effectively turns docker-compose into a preprocessor.

Offensive

Hi @bmitch. I know docker stack deploy does not have all features that docker-compose has. That what I wrote below in my own answer, with a link to the relative issue in GitHub. By the way, I didn't know we can use docker-compose as a preprocessor. Thanks for the tip. – Longshore

Thanks! I use a script deploy.sh that is simple and it just works: #!/usr/bin/env bash docker-compose config > docker-compose-parsed.yaml docker stack deploy -c ./docker-compose-parsed.yaml MY_SERVICE – Told

This should be the accepted answer -- use this in conjunction with docker stack deploy -c - which reads from stdin, so you have docker-compose config | docker stack deploy -c - – Hammel

This does not work anymore as compose files formats for Swarm and Compose now mismatch. – Shopping

Answer 3

The answer is quite simple: it's not a bug, nor a feature. .env is not currently supported by docker stack. You must source manually the .env running export $(cat .env) before running docker stack ...

There is an issue discussing this needs in the Docker Github. https://github.com/docker/docker.github.io/issues/3654 and another one discussing the problem and solution: https://github.com/moby/moby/issues/29133#issuecomment-285980447

Longshore

Answer 4

you can create a deploy.sh

export $(cat .env) > /dev/null 2>&1; docker stack deploy ${1:-STACK_NAME}

• The .env parses without regular expressions or unstable tricks.
• Errors on stderr produced by #comments inside the .env will be redirected to stdin (2>&1)
• Undesired prints of all export and error now on stdin too, are redirected to /dev/null. This prevents console flood.
• Those errors do not prevent .env being parsed correctly.

We can define STACK_NAME in our .env but we can also pass our customized stack_name

. deploy.sh <stack_name> (stack_name opcional)

This workaround gave me headaches for 3 nights

Authorship

This is what worked for me, simple solution to inject the .env content. Thanks. – Quesnay

Answer 5

I have encountered the same issue, My issue was caused by running docker-compose in a different python virtual environment.

I hope this could help.

It seems that by running on a different python virtual env, I ruined the docker-compose dependency pythondotenv

Defence

Answer 6

In 2024 this is how it works with env_file and environment variables including the interpolation.

Summary:

• create the deploy.sh script transforming raw your-service.swarm.yml into processed.swarm.yml with docker stack config command, then calling the docker stack deploy command on processed.swarm.yml.
• provide the env variables for the stack deploy script via the following command env $(cat your-service.stage.env | xargs) ./deploy.sh (e.g. with env-and-deploy.sh script)

This requires some example code to understand:

# file: your-service.swarm.yml
version: '3.12'
services:
  your-service:
    image: your-service-image
    env_file:
      - your-service.env
    environment:
      - SOME_ENV=qwerty
      - OTHER_ENV=${SOME_ENV}

# file: your-service.env
ANOTHER_VALUE=another-value
ANOTHER_ENV_VAR=${ANOTHER_VALUE}

# file: deploy.sh
#!/bin/bash
processed="processed.swarm.yml"
docker stack config --compose-file your-service.swarm.yml > "$processed"
docker stack deploy --compose-file "$processed" your-stack

# file: env-and-deploy.sh
#!/bin/bash
env $(cat your-service.stage.env | xargs) ./deploy.sh

Look up how the content of processed.swarm.yml and the running service container env list look like.

Shopping