How to set Docker Compose `env_file` relative to `.yml` file when multiple `--file` option is used?

Question

I am trying to set my env_file configuration to be relative to each of the multiple docker-compose.yml file locations instead of relative to the first docker-compose.yml.

The documentation (https://docs.docker.com/compose/compose-file/compose-file-v3/#env_file) suggests this should be possible:

If you have specified a Compose file with docker-compose -f FILE, paths in env_file are relative to the directory that file is in.

For example, when I issue

docker compose \
  --file docker-compose.yml \
  --file backend/docker-compose.yml \
  --file docker-compose.override.yml up

all of the env_file paths in the second (i.e. backend/docker-compose.yml) and third (i.e. docker-compose.override.yml) are relative to the location of the first file (i.e. docker-compose.yml)

I would like to have the env_file settings in each docker-compose.yml file to be relative to the file that it is defined in.

Is this possible?

Thank you for your time 🙏

In case you are curious about the context:

I would like to have a backend repo that is self-contained and the backend developer can just work on it without needing the frontend container. The frontend repo will pull in the backend repo as a Git submodule, because the frontend container needs the backend container as a dependency. Here are my 2 repo's:

• backend: https://gitlab.com/starting-spark/porter/backend
• frontend: https://gitlab.com/starting-spark/porter/frontend

The backend is organized like this:

/docker-compose.yml
/docker-compose.override.yml

The frontend is organized like this:

/docker-compose.yml
/docker-compose.override.yml
/backend/ # pulled in as a Git submodule
/backend/docker-compose.yml
/backend/docker-compose.override.yml

Everything works if I place my env_file inside the docker-compose.override.yml file. The backend's override env_file will be relative to the backend docker-compose.yml. And the frontend's override env_file will be relative to the frontend docker-compose.yml. The frontend will never use the backend's docker-compose.override.yml.

But I wanted to put the backend's env_file setting in to the backend's docker-compose.yml instead, so that projects needing the backend container can inherit and just use it's defaults. If the depender project wants to override backend's env_file, then it can do so in the depender project's docker-compose.override.yml.

I hope that makes sense.

If there's another pattern to organizing Docker-Compose projects that handles this scenario, please let me know.

• I did want to avoid a mono-repo.

Granvillegranvillebarker

The documentation makes it sound like it's supposed to work the way you want it to. That's the way I'd want it to work too, so it seems odd that it isn't working that way... – Hydropic

Answer 1

It turns out that there's already an issue and discussion regarding this:

• https://github.com/docker/compose/issues/3874

The thread points out that this is the expected behavior and is documented here: https://docs.docker.com/compose/extends/#understanding-multiple-compose-files

When you use multiple configuration files, you must make sure all paths in the files are relative to the base Compose file (the first Compose file specified with -f). This is required because override files need not be valid Compose files. Override files can contain small fragments of configuration. Tracking which fragment of a service is relative to which path is difficult and confusing, so to keep paths easier to understand, all paths must be defined relative to the base file.

There's a workaround within that discussion that works fairly well: https://github.com/docker/compose/issues/3874#issuecomment-470311052

The workaround is to use a ENV var that has a default:

• ${PROXY:-.}/haproxy/conf:/usr/local/etc/haproxy

Or in my case:

  env_file:
    - ${BACKEND_BASE:-.}/.env

Hope that can be helpful for others 🤞

In case anyone is interested in the full code: backend's docker-compose.yml: https://gitlab.com/starting-spark/porter/backend/-/blob/3.4.3/docker-compose.yml#L13-14
backend's docker-compose.override.yml: https://gitlab.com/starting-spark/porter/backend/-/blob/3.4.3/docker-compose.override.yml#L3-4
backend's .env: https://gitlab.com/starting-spark/porter/backend/-/blob/3.4.3/.env
frontend's docker-compose.yml: https://gitlab.com/starting-spark/porter/frontend/-/blob/3.2.2/docker-compose.yml#L5-6
frontend's docker-compose.override.yml: https://gitlab.com/starting-spark/porter/frontend/-/blob/3.2.2/docker-compose.override.yml#L3-4
frontend's .env: https://gitlab.com/starting-spark/porter/frontend/-/blob/3.2.2/.env#L16

Granvillegranvillebarker