How to run docker-compose inside docker in docker which runs inside gitlab-runner container?

Question

I have a gitlab runner inside a docker container, runs fine if I run an image like nginx. But now I tried to run docker in docker (dind) inside the gitlab runner and I want to run docker-compose inside the dind. Docker info runs fine, but if I try to run docker-compose I get an permission-denied error.

I linked the /usr/local/bin/docker-compose file to the gitlab runner container and enter it in the volumes parameter in the runner config.toml file.

If I try to run sudo it ends with an unknown command error, so that could not be the solution.

Do I have to link some file more or are the to many nested containers?

Linkoski

Answer 1

if you are using dind it means docker is working OK, now you just have to install docker-compose that is just simple python package and you can do it in before_script

.gitlab-ci.yml

image: docker:latest

services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2

stages:
- test

before_script:
  - apk add --no-cache py-pip
  - pip install docker-compose
  - docker info
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN docker.registry.com

test:
  stage: test
  script:

    - cp .env.sample .env # copy environement variable
    - docker-compose up -d
    # run some test here

Kiwi

The command pip install docker-compose throws error: command 'gcc' failed with exit status 1 – Bewail

Sorry to bring this up after 2 years but within "script" I can see my shell environment variables but they are not populated in the docker-compose yml. Do you understand why this is happening? Thank you in advance. – Nailhead

@PierreCordier, this needs the dependencies plus rust cargo from the other answer before pip install compose - the dependencies are apk add py-pip python3-dev libffi-dev openssl-dev gcc libc-dev make rust cargo see the yellow message about alpine at this link under the linux tab – Datnow

Answer 2

In order to have docker-compose, you need to install it for image docker, which is of version 18.09.6, build 481bc77 at the time of writing.

Since docker-compose version 1.24.0, you also need the following dependencies for docker-compose to be installed on alpine:

apk add py-pip python3-dev libffi-dev openssl-dev gcc libc-dev make

Here is a sample .gitlab-ci.yml:

image: docker:stable

stages:
    - deploy

services:
  - docker:dind

before_script:
  - apk update
  - apk add py-pip python3-dev libffi-dev openssl-dev gcc libc-dev make
  - pip install docker-compose

deploy_app:
    stage: deploy
    script:
        - docker-compose down
        - docker-compose up -d

Watermelon

Absolutely terrible .gitlab-ci.yml without version pinning of the docker & dind images! Instead of writing the version in text, why not pin it all in config? – Erse

@gtxbxakgcanmt9d9 The SO philosophy is to show, not tell. An example would be fantastic :-) – Proscription

Answer 3

The best choice for running docker-compose inside docker in docker is using the docker/compose image as bellow:

job-x:
  image: docker/compose
  script:
    - docker-compose version

A real and full example of this is using docker-compose to deploy with specific installed runner on a server as bellow:

image: docker:stable

services:
  - docker:dind

stages:
  - build
  - deploy

before_script:
  # resolve TAG
  # Default branch leaves tag latest
  # All other branches are tagged with the escaped branch name (commit ref slug)
  - |
    if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
      TAG="latest"
      echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = '$TAG'"
    else
      TAG="$CI_COMMIT_REF_SLUG"
      echo "Running on branch '$CI_COMMIT_BRANCH': tag = $TAG"
    fi
  

docker-build:
  stage: build
  script:
    - echo "user ${CI_REGISTRY_USER}"
    - echo "registry ${CI_REGISTRY}"
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$CI_REGISTRY_IMAGE:${TAG}" -t "$CI_REGISTRY_IMAGE:${TAG}" .
    - docker push "$CI_REGISTRY_IMAGE:${TAG}"
  when: manual

deploy-staging:
  image: docker/compose
  stage: deploy
  variables:
    IMAGE: $CI_REGISTRY_IMAGE
    TAG: $TAG
    HOST_PORT: $HOST_PORT
  script:
    - docker-compose version
    - docker-compose -f docker-compose.yml up -d --no-build
  when: manual

The docker-compose.yml file seems like:

version: "3.7"

services:
  web:
    image: $IMAGE:$TAG
    build:
      context: .
    ports:
      - $HOST_PORT:8000

Deepen