Run docker-compose build in .gitlab-ci.yml

Asked 5/10, 2016 at 7:56 Answered 7/5 at 13:23

109

I have a .gitlab-ci.yml file which contains following:

image: docker:latest

services:
  - docker:dind

before_script:
  - docker info
  - docker-compose --version

buildJob:
  stage: build
  tags:
    - docker
  script:
    - docker-compose build

But in ci-log I receive message:

$ docker-compose --version
/bin/sh: eval: line 46: docker-compose: not found

What am I doing wrong?

Epithelium answered 5/10, 2016 at 7:56 Comment(0)

Deprecated

docker-compose can be installed as a Python package, which is not shipped with your image. The image you chose does not even provide an installation of Python:

$ docker run --rm -it docker sh
/ # find / -iname "python"
/ #

Looking for Python gives an empty result. So you have to choose a different image, which fits to your needs and ideally has docker-compose installed or you maually create one.

The docker image you chose uses Alpine Linux. You can use it as a base for your own image or try a different one first if you are not familiar with Alpine Linux.

I had the same issue and created a Dockerfile in a public GitHub repository and connected it with my Docker Hub account and chose an automated build to build my image on each push to the GitHub repository. Then you can easily access your own images with the GitLab CI.

Periderm answered 5/10, 2016 at 8:31 Comment(2)

Can you delete one of your answers? Confusing to find two, plus lots of "EDIT" messages. (We can see edit history if we need to). – Spondylitis 10/6, 2019 at 8:35

I cannot delete an accepted answer :-/ – Periderm 14/3, 2023 at 8:39

117

EDIT (2023): Do not keep using docker-compose! Please migrate to docker compose, which is now part of the docker image and a 99% drop-in replacement of docker-compose. Ironically, if you (now) use docker:latest, the question does not apply. See Docker's documentation on Compose V2. The answer below only applies to docker-compose, AKA Compose V1, which is to be discontinued/become unsupported.

Docker also provides an official docker-compose image: docker/compose

This is the ideal solution if you don't want to install it every pipeline.

Note that in the latest version of GitLab CI/Docker you will likely need to give privileged access to your GitLab CI Runner and configure/disable TLS. See Use docker-in-docker workflow with Docker executor

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

# Official docker compose image.
image:
  name: docker/compose:latest

services:
  - docker:dind

before_script:
  - docker version
  - docker-compose version

build:
  stage: build
  script:
    - docker-compose down
    - docker-compose build
    - docker-compose up tester-image

Note that in versions of docker-compose earlier than 1.25:

Since the image uses docker-compose-entrypoint.sh as entrypoint you'll need to override it back to /bin/sh -c in your .gitlab-ci.yml. Otherwise your pipeline will fail with No such command: sh

    image:
      name: docker/compose:latest
      entrypoint: ["/bin/sh", "-c"]

Inflammation answered 10/10, 2018 at 6:34 Comment(13)

This works, but as your linked source says, it will need a GitLab *9.4**+ , because you **need to overwrite the entrypoint in the CI yaml. – Truss 24/10, 2018 at 9:58

@Inflammation I'm getting this error when the execution reaches "docker-compose version"...

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ERROR: Job failed: exit code 1

... how can I solve it? – Couplet 27/10, 2018 at 5:28

It only worked when I added the variables DOCKER_HOST: tcp://docker:2375/ and DOCKER_DRIVER: overlay2 – Couplet 27/10, 2018 at 10:6

That depends on your docker/runner setup. See docs.gitlab.com/ee/ci/docker/using_docker_build.html There are various ways to set up the runner, so you should add any needed config where necessary. – Inflammation 29/10, 2018 at 13:49

About the need to override the entrypoint, I opened an issue at github.com/docker/compose/issues/6614, some +1 could help to make it fix! thanks you. – Harte 31/3, 2019 at 17:34

It works! If bind mounts or volumes are involved, some environment variables are needed as well -- DOCKER_HOST=tcp://docker:2375 and DOCKER_DRIVER=overlay2 (optional) -- to instruct docker to talk with the daemon started inside of the service (docker:dind). Refer to Use docker-in-docker executor for more information. – Scoundrelly 11/5, 2019 at 3:50

This still doesn't work for me (shared gitlab.com runners) even when all variables are set, Cannot connect to the Docker daemon at tcp://docker:2375/. Is the docker daemon running? – Tweezers 12/6, 2020 at 13:58

@TheGodfather Probably because of TLS restrictions. See docs.gitlab.com/ee/ci/docker/… – Inflammation 7/7, 2020 at 15:22

@Inflammation I did read this, but that's why I highlighted - shared Gitlab.com runners. What they provide. I have nothing to do with Docker version, TLS or whatsoever, I'm using shared runners and I expect them to work out-of-the-box. – Tweezers 7/7, 2020 at 16:29

I haven't used gitlab.com shared runners, but from what I can see here they've disabled tls by default, but do mount /certs/client for some reason. Do you have the service definition docker:dind in your (job) config? And yes you do have to comply with whatever they are using, and properly run docker-in-docker, otherwise it will not work. Also, the dind version and docker-compose version should be compatible. Using "latest" tags is not recommended. – Inflammation 10/7, 2020 at 7:16

With the shell executor, it worked without any environment variables, but my sysadmin had a custom MTU, so I had to change in /etc/docker/daemon.json the mtu value. – Dacoity 4/3, 2021 at 10:58

The docker/compose image is deadware. Hasn't recieved updates since 2 years ago. It still features docker 19.03 and the deprecated compose V1. – Sumptuous 13/3, 2023 at 19:37

@Sumptuous that is a valid point, but I'm guessing users of compose v2 don't need this, because it is part of the docker image now (docker-compose has been replaced with docker compose) see docs.docker.com/compose/compose-v2 – Inflammation 14/3, 2023 at 20:59

Following the official documentation:

# .gitlab-ci.yml
image: docker
services:
  - docker:dind    
build:
  script:
    - apk add --no-cache docker-compose
    - docker-compose up -d

Sample docker-compose.yml:

version: "3.7"
services:
  foo:
    image: alpine
    command: sleep 3
  bar:
    image: alpine
    command: sleep 3

We personally do not follow this flow anymore, because you loose control about the running containers and they might end up running endless. This is because of the docker-in-docker executor. We developed a python-script as a workaround to kill all old containers in our CI, which can be found here. But I do not suggest to start containers like this anymore.

Periderm answered 9/3, 2017 at 14:24 Comment(9)

@tmaier I think i am stuck here too. What kind of "Runner-Installation" did you use? I mean which of the three ways did you go? docs.gitlab.com/ee/ci/docker/… – Xerox 8/5, 2017 at 9:23

We are using the docker-in-docker executor docs.gitlab.com/ee/ci/docker/… – Periderm 8/5, 2017 at 12:27

While this works, it means that on every build docker-compose needs to be installed (and fetched...) and thus it will increase the build time. If you have multiple jobs needing docker-compose (and in a production scenario you must likely will), build time will considerably expand. – Truss 12/9, 2018 at 0:19

If you want to use it more frequently, you can define your own docker image für your CI with a small Dockerfile containing online three lines: ` FROM alpine \n RUN apk add --no-cache py-pip && pip install docker-compose ` But keep in mind to keep this image always up-to-date – Periderm 12/9, 2018 at 10:21

Thank you very much for your answer. I just have an issue with the networking. It seems no container can communicate with another… Am I the only one in this situation? (using shared runners) – Goodsell 18/10, 2018 at 12:32

This solution is no longer working for me as of now. However alpine linux now has a docker-compose package in their edge branch, which I think is the way to go. – Frightful 9/4, 2019 at 10:22

regarding your "Update from 2019-04-09", maybe you should also use docker-compose down to terminate all running services, after your CI task is done? – Inflammation 25/4, 2019 at 6:4

try: services: - docker:18-dind – Futtock 6/8, 2019 at 6:30

"because you loose control about the running containers". Why not? Can't you do docker-compose down? – Harewood 9/10, 2022 at 12:24

Deprecated

I created a simple docker container which has docker-compose installed on top of docker:latest. See https://hub.docker.com/r/tmaier/docker-compose/ or https://github.com/tmaier/docker-compose

Your .gitlab-ci.yml file would look like this:

image: tmaier/docker-compose:latest

services:
  - docker:dind

before_script:
  - docker info
  - docker-compose --version

buildJob:
  stage: build
  tags:
    - docker
  script:
    - docker-compose build

Pillbox answered 4/1, 2017 at 1:49 Comment(1)

getting this error in gitlab ci error: command 'gcc' failed with exit status 1 – Laceration 7/7, 2020 at 18:27

Deprecated

docker-compose can be installed as a Python package, which is not shipped with your image. The image you chose does not even provide an installation of Python:

$ docker run --rm -it docker sh
/ # find / -iname "python"
/ #

Looking for Python gives an empty result. So you have to choose a different image, which fits to your needs and ideally has docker-compose installed or you maually create one.

The docker image you chose uses Alpine Linux. You can use it as a base for your own image or try a different one first if you are not familiar with Alpine Linux.

Periderm answered 5/10, 2016 at 8:31 Comment(2)

Can you delete one of your answers? Confusing to find two, plus lots of "EDIT" messages. (We can see edit history if we need to). – Spondylitis 10/6, 2019 at 8:35

I cannot delete an accepted answer :-/ – Periderm 14/3, 2023 at 8:39

If you don't want to provide a custom docker image with docker-compose preinstalled, you can get it working by installing Python during build time. With Python installed you can finally install docker-compose ready for spinning up your containers.

image: docker:latest

services:
- docker:dind

before_script:
- apk add --update python py-pip python-dev && pip install docker-compose # install docker-compose
- docker version
- docker-compose version

test:
  cache:
    paths:
    - vendor/
  script:
  - docker-compose up -d
  - docker-compose exec -T php-fpm composer install --prefer-dist
  - docker-compose exec -T php-fpm vendor/bin/phpunit --coverage-text --colors=never --whitelist src/ tests/

Use docker-compose exec with -T if you receive this or a similar error:

$ docker-compose exec php-fpm composer install --prefer-dist
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 9, in <module>
    load_entry_point('docker-compose==1.8.1', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 62, in main
    command()
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 114, in perform_command
    handler(command, command_options)
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 442, in exec_command
    pty.start()
  File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
    io.set_blocking(pump, flag)
  File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
    old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)
ERROR: Build failed: exit code 1

Mouthwash answered 2/11, 2016 at 18:27 Comment(3)

Not a common way, actually you can get "apk: command not found" – Bamako 6/4, 2019 at 6:50

this no longer works, possibly because of the newer docker images. now python and python-dev are no longer valid apk package adds, and then when you specify python2, gcc is missing somehow – Laceration 7/7, 2020 at 18:23

Outdated. Official docker images already feature compose V2. Instead, this would shadow that with the deprecated compose V1 – Sumptuous 14/3, 2023 at 7:27

there is tiangolo/docker-with-compose which works:

image: tiangolo/docker-with-compose

stages:
  - build
  - test
  - release
  - clean

  
before_script:
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com

build:
  stage: build
  script:
    - docker-compose -f docker-compose-ci.yml build --pull 


test1:
    stage: test
    script:
        - docker-compose -f docker-compose-ci.yml up -d
        - docker-compose -f docker-compose-ci.yml exec -T php ...

Fundamentalism answered 2/11, 2020 at 22:19 Comment(1)

This does not work for me. I get "error during connect: Post docker:2375/v1.40/auth: dial tcp: lookup docker on 169.254.169.254:53: no such host". webmaster777's solution, however, worked for me and seems like the best solution as the main reason tiangolo/docker-with-compose was created was because docker/compose previously had an entrypoint problem. Now that that is not an issue, I think this might no longer be a good solution. – Montserrat 7/5, 2021 at 10:15

I think most of the above are helpful, however i needed to collectively apply them to solve this problem, below is the script which worked for me

I hope it works for you too

Also note, in your docker compose this is the format you have to provide for the image name

<registry base url>/<username>/<repo name>/<image name>:<tag>

image:
  name: docker/compose:latest
  entrypoint: ["/bin/sh", "-c"]

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

services:
  - docker:dind

stages:
- build_images

before_script:
  - docker version
  - docker-compose version
  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY

build:
  stage: build_images
  script:
    - docker-compose down
    - docker-compose build
    - docker-compose push

Bohemian answered 9/3, 2020 at 4:11 Comment(2)

The docker/compose image is deadware, no updates for 2 years and still featuring docker 19 and the deprecated compose V1 – Sumptuous 14/3, 2023 at 7:25

Never said docker-compose was dead, but the docker/compose image is. No hard feelings, just trying to signal that an answer should be avoided. The answer was good advice 3 years ago, just not anymore. – Sumptuous 21/3, 2023 at 10:26

It really took me some time to get it working with Gitlab.com shared runners.

I'd like to say "use docker/compose:latest and that's it", but unfortunately I was not able to make it working, I was getting Cannot connect to the Docker daemon at tcp://docker:2375/. Is the docker daemon running? error even when all the env variables were set.

Neither I like an option to install five thousands of dependencies to install docker-compose via pip.

Fortunately, for the recent Alpine versions (3.10+) there is docker-compose package in Alpine repository. It means that @n2o's answer can be simplified to:

test:
  image: docker:19.03.0

  variables:
    DOCKER_DRIVER: overlay2
    # Create the certificates inside this directory for both the server
    # and client. The certificates used by the client will be created in
    # /certs/client so we only need to share this directory with the
    # volume mount in `config.toml`.
    DOCKER_TLS_CERTDIR: "/certs"

  services:
    - docker:19.03.0-dind

  before_script:
    - apk --no-cache add docker-compose      # <---------- Mind this line
    - docker info
    - docker-compose --version

  stage: test
  script:
      - docker-compose build

This worked perfectly from the first try for me. Maybe the reason other answers didn't was in some configuration of Gitlab.com shared runners, I don't know...

Tweezers answered 12/6, 2020 at 14:18 Comment(2)

This is really the solution on 2021, all others didn't work with a custom runner – Fledgling 12/3, 2021 at 14:23

This installs the deprecated compose V1 – Sumptuous 14/3, 2023 at 7:24

As a consequence of https://github.com/docker-library/docker/pull/361 just do

# docker-compose.yml
version: '2'
services:
  hello:
    image: hello-world

# .gitlab-ci.yml
myjob:
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker --version
    - docker compose version

If you are not using the official docker "docker" image, do not install compose with a package manager as most distributions are outdated. Official install instructions point to either use the downloads.docker.com repository in its diverse forms for each Linux distribution or to simply download the github release artifact, which I think is leaner for a CI usage:

myjob:
    - mkdir -p ~/.docker/cli-plugins
    - (wget --no-verbose --no-clobber https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 --output-document ~/.docker/cli-plugins/docker-compose || true) # ideally you would cache this binary
    - chmod a+x ~/.docker/cli-plugins/docker-compose
    - ln --symbolic ~/.docker/cli-plugins/docker-compose /usr/local/bin/docker-compose
  script:
    - docker-compose version

Sumptuous answered 14/3, 2023 at 11:31 Comment(0)

Updated 2024 - After trying any option, currently it works properly (Tested)

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

stages:
  - build

before_script:
  - docker version
  - docker-compose version

build_image:
  image: docker:20-dind
  stage: build
  services:
    - name: docker:20-dind
      alias: docker
      command: ["--tls=false"]

  script:
    # It takes your registry as pipe and protect user and password by default
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY

    - docker-compose up -d --build
    - docker push $CI_REGISTRY/group/project/image:version
    - echo "Image built successfully"
    - docker images

NOTE: Replace your current /group/project/image:version path checking gitlab repository in repository > packages and registries > container registry section, and also add variables if you need globally in project > settings > variables or in variables inside your .gitlab-ci.yml instead.

Spigot answered 7/5 at 13:23 Comment(0)

-1

Alpine linux now has a docker-compose package in their "edge" branch, so you can install it this way in .gitlab-ci.yml


a-job-with-docker-compose:
  image: docker
  services:
    - docker:dind
  script:
    - apk add docker-compose --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
    - docker-compose -v

Frightful answered 9/4, 2019 at 10:25 Comment(2)

not working.. needs to update answer.. When run docker-compose throws

pkg_resources.DistributionNotFound: The 'docker-compose==1.24.0' distribution was not found and is required by the application

– Ubangi 20/6, 2019 at 18:54

This installs the deprecated compose V1 – Sumptuous 14/3, 2023 at 7:24

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags