How to keep docker image build during job across two stages with Gitlab CI?

Question

I use Gitlab runner on an EC2 to build, test and deploy docker images on a ECS.

I start my CI workflow using a "push/pull" logic: I build all my docker images during the first stage and push them to my gitlab repository then I pull them during the test stage.

I thought that I could drastically improve the workflow time by keeping the image built during the build stage between build and test stages.

My gitlab-ci.yml looks like this:

stages:
  - build
  - test
  - deploy

build_backend:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t backend:$CI_COMMIT_BRANCH ./backend
  only:
    refs:
      - develop
      - master

build_generator:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t generator:$CI_COMMIT_BRANCH ./generator
  only:
    refs:
      - develop
      - master

build_frontend:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t frontend:$CI_COMMIT_BRANCH ./frontend
  only:
    refs:
      - develop
      - master

build_scraping:
  stage: build
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  script:
    - docker build -t scraping:$CI_COMMIT_BRANCH ./scraping
  only:
    refs:
      - develop
      - master


test_backend:
  stage: test
  needs: ["build_backend"]
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
    - mkdir -p $DOCKER_CONFIG/cli-plugins
    - apk add curl
    - curl -SL https://github.com/docker/compose/releases/download/v2.3.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
    - chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
  script:
    - docker compose -f docker-compose-ci.yml up -d backend
    - docker exec backend pip3 install --no-cache-dir --upgrade -r requirements-test.txt
    - docker exec db sh mongo_init.sh
    - docker exec backend pytest test --junitxml=report.xml -p no:cacheprovider
  artifacts:
    when: always
    reports:
      junit: backend/report.xml
  only:
    refs:
      - develop
      - master

test_generator:
  stage: test
  needs: ["build_generator"]
  image: docker
  services:
    - docker:dind
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
    - mkdir -p $DOCKER_CONFIG/cli-plugins
    - apk add curl
    - curl -SL https://github.com/docker/compose/releases/download/v2.3.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
    - chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
  script:
    - docker compose -f docker-compose-ci.yml up -d generator
    - docker exec generator pip3 install --no-cache-dir --upgrade -r requirements-test.txt
    - docker exec generator pip3 install --no-cache-dir --upgrade -r requirements.txt
    - docker exec db sh mongo_init.sh
    - docker exec generator pytest test --junitxml=report.xml -p no:cacheprovider
  artifacts:
    when: always
    reports:
      junit: generator/report.xml
  only:
    refs:
      - develop
      - master
   
[...]

gitlab-runner/config.toml:

concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Docker Runner"
  url = "https://gitlab.com/"
  token = ""
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.12"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0

docker-compose-ci.yml:

services:
  backend:
    container_name: backend
    image: backend:$CI_COMMIT_BRANCH
    build:
      context: backend
    volumes:
      - ./backend:/app
    networks:
      default:
    ports:
      - 8000:8000
      - 587:587
      - 443:443
    environment:
      - ENVIRONMENT=development
    depends_on:
      - db

  generator:
    container_name: generator
    image: generator:$CI_COMMIT_BRANCH
    build:
      context: generator
    volumes:
      - ./generator:/var/task
    networks:
      default:
    ports:
      - 9000:8080
    environment:
      - ENVIRONMENT=development
    depends_on:
      - db

  db:
    container_name: db
    image: mongo
    volumes:
      - ./mongo_init.sh:/mongo_init.sh:ro
    networks:
      default:
    environment:
      MONGO_INITDB_DATABASE: DB
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: admin
    ports:
      - 27017:27017

  frontend:
    container_name: frontend
    image: frontend:$CI_COMMIT_BRANCH
    build:
      context: frontend
    volumes:
      - ./frontend:/app
    networks:
      default:
    ports:
      - 8080:8080
    depends_on:
      - backend

networks:
  default:
    driver: bridge

When I comment context: in my docker-compose-ci.yml, Docker can't find my image and indeed it is not keep between jobs.

What is the best Docker approach during CI to build -> test -> deploy? Should I zip my docker image and share them between stages using artifacts? It doesn't seem to be the most efficient way to do this.

I'm a bit lost about which approach I should use to perform a such common workflow in Gitlab CI using Docker.

Reina

Answer 1

The best way to do this is to push the image to the registry and pull it in other stages where it is needed. You appear to be missing the push/pull logic.

You also want to make sure you've leveraging docker caching in your docker builds. You'll probably want to specify the cache_from: key in your compose file.

For example:

build:
  stage: build
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # pull latest image to leverage cached layers
    - docker pull $CI_REGISTRY_IMAGE:latest || true

    # build and push the image to be used in subsequent stages
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA  # push the image

test:
  stage: test
  needs: [build]
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # pull the image that was built in the previous stage
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker-compose up # or docker run or whatever

---

Edit:

In modern versions of Docker with Docker buildkit/buildx, you can use the buildkit inline caching instead of pulling the image ahead of time. This requires pushing a somewhat larger image to your repo, but makes cache pulls speedier because docker can tell which layers are valid for caching before it pulls them. Normal pull speed is unaffected.

export DOCKER_BUILDKIT=1

docker build --build-arg BUILDKIT_INLINE_CACHE=1 \
             --cache-from "$CI_REGISTRY_IMAGE:latest"
             --cache-from "$CI_REGISTRY_IMAGE:$CI_REF_NAME" \
             --tag "$CI_REGISTRY_IMAGE:$CI_REF_NAME" \
             --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" \ 
             .

docker push "$CI_REGISTRY_IMAGE:$CI_REF_NAME"
docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Intersidereal

Thank you, I will keep the push & pull logic I wrote before so. I will use docker caching as you explain to reduce my build time, it's a good idea. – Reina

Just trying to understand, you are still pulling the from-cache image, how will this save time? The op wanted a way to not pull the images – Guidotti

@RakeshGupta OP was having an issue because they ended up (unexpectedly) re-building the image in the test stage instead of re-using the already built image in the build stage. OP was expecting the docker build process to have been cached. So, this answer solves that problem. Pulling the image to leverage cached layers is almost always much faster than building (and re-building) it. – Intersidereal

Pushing and pulling is not a good solution if you want to keep your login data off non protected branches. Here you need CI_REGISTRY_PASSWORD to be available in non protected branches. Like this you cannot check the Protected Variable flag for CI_REGISTRY_PASSWORD. – Majormajordomo

You can also use docker save and docker load in conjunction with artifacts instead. – Majormajordomo

Answer 2

Try mounting the "Docker Root Dir" as a persistent/nfs volume that is shared by the fleet of runners.

Docker images are stored in "Docker Root Dir" path. You can find out your docker root by running:

# docker info
...
 Storage Driver: overlay2
 Docker Root Dir: /var/lib/docker
...

Generally the default paths based on the OS are

Ubuntu: /var/lib/docker/
Fedora: /var/lib/docker/
Debian: /var/lib/docker/
Windows: C:\ProgramData\DockerDesktop
MacOS: ~/Library/Containers/com.docker.docker/Data/vms/0/

Once properly mounted to all agents, you will be able to access all local docker images.

References:

https://docs.gitlab.com/runner

https://blog.nestybox.com/2020/10/21/gitlab-dind.html

Guidotti

Keep in mind, this can be a security issue if the runner handles builds of multiple projects. The performance hit this would introduce for docker storage also makes this questionable, in my mind. Also, /var/lib/docker is only ever intended to be access from one daemon at any given time... using this concurrently across multiple hosts could result in inconsistencies (for example, one daemon could remove an image that's in use by a container running on another daemon because the first daemon has no idea it is in use by a container) – Intersidereal

@Rakesh Gupta it is indeed the solution I have in mind, thank you! – Reina

@Intersidereal Thank you too for your advice, during my parallels jobs I never pull the same container image and I use this CI/CD only for one project so it should be good, no? – Reina

@Reina from a security perspective, it wouldn't be an issue if your runner host is only used for one project. However, there's still a data corruption risk from concurrent access to /var/lib/docker by multiple daemons. – Intersidereal