TestContainers integration test on Gitlab CI

Question

I have java project which using testcontainers for integration test. I want to implement gitlab ci for that stage but I've got that error

java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration

The project build successfully on local machine but gitlap ci docker couldn't start propeply

Here a log output

[0KRunning with gitlab-runner 14.3.0 (b37d3da9)[0;m
[0K  on Backend Docker runner 9L3Zko1w[0;m
section_start:1657698628:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m[0;m
[0KUsing Docker executor with image maven:3.6.0-jdk-11-slim ...[0;m
[0KStarting service docker:dind ...[0;m
[0KPulling docker image docker:dind ...[0;m
[0KUsing docker image sha256:232342342342 for docker:dind with digest docker@sha256:2342342342 ...[0;m
[0KWaiting for services to be up and running...[0;m

[0;33m*** WARNING:[0;m Service runner-23423423-docker-0 probably didn't start properly.

Health check error:
service "runner-234234234-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2022-07-13T07:50:40.800780292Z ip: can't find device 'ip_tables'
2022-07-13T07:50:40.801898214Z ip_tables              32768  2 iptable_filter,iptable_nat
2022-07-13T07:50:40.802546323Z x_tables               45056  5 xt_conntrack,xt_MASQUERADE,xt_addrtype,iptable_filter,ip_tables
2022-07-13T07:50:40.802609399Z modprobe: can't change directory to '/lib/modules': No such file or directory
2022-07-13T07:50:40.806461330Z mount: permission denied (are you root?)
2022-07-13T07:50:40.807019652Z Could not mount /sys/kernel/security.
2022-07-13T07:50:40.807028026Z AppArmor detection and --privileged mode might break.
2022-07-13T07:50:40.807931221Z mount: permission denied (are you root?)

here is gitlab ci yaml

services:
  - name: docker:dind
    # explicitly disable tls to avoid docker startup interruption
    command: ["--tls=false"]
variables:
  # Instruct Testcontainers to use the daemon of DinD.
  DOCKER_HOST: "tcp://docker:2375"
  # Instruct Docker not to start over TLS.
  DOCKER_TLS_CERTDIR: ""
  # Improve performance with overlayfs.
  DOCKER_DRIVER: overlay2
test:
  image: maven:3.8.6-jdk-11-slim
  stage: ⚙️ maven-build
  before_script:
    - docker info
    - cp $MAVEN_SETTINGS_XML ~/.m2/settings.xml
  script:
    - mvn $MAVEN_CLI_OPTS clean verify

Frequentative

What kind of Gitlab setup is this? Public cloud? On premise? Which executor? – Emmerich

I am using public gitlab – Frequentative

i ran into the same issue, i am trying a solution is to specify docker:${version}-dind so i use docker:24.0.6-dind, i will monitor the build to see if the issue rises again – Inclusive

Answer 1

Just using the maven:3.8.6-jdk-11-slim image is not enough. You also need docker installed in your image. A very easy way to get docker is by just installing it manually in your pipeline:

before_script:
    - curl -fsSL https://get.docker.com -o get-docker.sh
    - bash ./get-docker.sh
    - docker info

However, it would be much better if you installed docker in a custom docker image so that you aren't constantly installing docker on every pipeline run. You can do that with this dockerfile:

FROM maven:3.8.6-jdk-11-slim
RUN curl -fsSL https://get.docker.com -o get-docker.sh
RUN bash ./get-docker.sh

The resulting image can then be uploaded to a registry and used instead of maven:3.8.6-jdk-11-slim without needing to install docker each time.

Waterman

You don't need Docker installed when using Testcontainers and having the dind service attached should normally be sufficient. – Emmerich

I've never been able to run docker commands with the dind service unless my base image also had docker installed – Waterman

For docker CLI commands this is correct. However, Testcontainers interacts with the Docker daemon over the Docker REST API. But you are right, the example above contains a call to docker info, which would require the CLI to be present. – Emmerich

This is still the only alternative ? I mean, the only way is to have an image with both docker and maven? – Eastwards

Any solution? I'm running into the same issue, and I'm not sure what's the correct approach – Peritoneum

Yes, it looks like this is the only way: to use image with dind and maven, or to use TestContainers Cloud. But they have only 300 minutes per month – Dacoity

Answer 2

Okay, after 4 days of looking for the solution, I finally want to share it here. I hope it will save you a lot of time and you won't be struggling with that.

1.Run gitlab runner in container

docker run -d --name gitlab-runner --restart always 
    -v ~/gitlab-runner/config:/etc/gitlab-runner \
    -v /var/run/docker.sock:/var/run/docker.sock \
    gitlab/gitlab-runner:latest

2.Register gitlab runner

docker run --rm -it -v ~/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

You will be suggested to write URL, token, and executor (it will be docker)

3.Go to ~/gitlab-runner/config/config.toml, and you will see something like that:

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "dind runner"
  url = "your_url"
  id = 6
  token = "your_token"
  token_obtained_at = 2024-01-26T17:46:43Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "docker:20.10.5"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0

I'm using docker:20.10.5 because with docker:latest I had that problem

can't open '/proc/net/ip6_tables_names': No such file or directory docker

the solution was found here

4.And the last step is gitlab-ci.yml

stages:
  - build
  - publish
  - run

variables:
  MAVEN_OPTS: >-
    -Dhttps.protocols=TLSv1.2
    -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository
    -Dorg.slf4j.simpleLogger.showDateTime=true
    -Djava.awt.headless=true
  MAVEN_CLI_OPTS: >-
    --batch-mode
    --errors
    --fail-at-end
    --show-version
    --no-transfer-progress

cache:
  paths:
    - .m2/repository

maven-job:
  image: intension/docker-dind-maven:3.8.5-r0-openjdk17
  stage: build

  variables:
    DOCKER_TLS_CERTDIR: ""

  script:
    - mvn --version  # check that mvn is working
    - docker images  # check that docker is working
    - mvn $MAVEN_CLI_OPTS clean package

Image in the job was one of the most important parts, that image contains maven + docker-dind

My env where it's working:
Gitlab: GitLab Community Edition v16.7.4
OS: Ubuntu 22.04.3 LTS

Dacoity