Cannot connect to the Docker daemon at unix:///var/run/docker.sock in gitlab CI

Question

I looked at any other questions but can't find my own solution! I setting up a CI in gitlab and use the gitlab's shared runner. In build stage I used docker image as base image but when i use docker command it says :

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I looked at this topic but still don't understand what should I do?

.gitlab-ci.yml :

stages:
  - test
  - build
  - deploy

job_1:
  image: python:3.6
  stage: test
  script:
    - sh ./sh_script/install.sh
    - python manage.py test -k

job_2:
  image: docker:stable
  stage: build
  before_script:
    - docker info
  script:
    - docker build -t my-docker-image .

I know that the gitlab runner must registered to use docker and share /var/run/docker.sock! But how to do this when using the gitlab own runner?

Patric

Answer 1

Ahh, that's my lovely topic - using docker for gitlab ci. The problem you are experiencing is better known as docker-in-docker.

Before configuring it, you may want to read this brilliant post: http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

That will give you a bit of understanding what is the problem and which solution best fits you. Generally there are 2 major approaches: actual installation of docker daemon inside docker and sharing host's daemon to containers. Which approach to choose - depends on your needs.

In gitlab you can go in several ways, I will just share our experience.

Way 1 - using docker:dind as a service.

It is pretty simple to setup. Just add docker:dind as a shared service to your gitlab-ci.yml file and use docker:latest image for your jobs.

image: docker:latest  # this sets default image for jobs
services:
  - docker:dind

Pros:

• simple to setup.
• simple to run - your source codes are available by default to your job in cwd because they are being pulled directly to your docker runner

Cons: you have to configure docker registry for that service, otherwise you will get your Dockerfiles built from scratch each time your pipeline starts. As for me, it is unacceptable, because can take more than an hour depending on the number of containers you have.

Way 2 - sharing /var/run/docker.sock of host docker daemon

We setup our own docker executor with docker daemon and shared the socket by adding it in /etc/gitlab-runner/config.toml file. Thus we made our machine's docker daemon available to docker cli inside containers. Note - you DONT need privileged mode for executor in this case.

After that we can use both docker and docker-compose in our custom docker images. Moreover, we dont need special docker registry because in this case we share executor's registry among all containers.

Cons

You need to somehow pass sources to your containers in this case, because you get them mounted only to docker executor, but not to containers, launched from it. We've stopped on cloning them with command like git clone $CI_REPOSITORY_URL --branch $CI_COMMIT_REF_NAME --single-branch /project

Joselow

In the second way, how can I access to the /etc/gitlab-runner/config.toml? – Patric

As I mentioned for the second approach, we use our own docker executor (VPS service). If you use provided by gitlab, you wont be able to share the socket, sorry – Joselow

How do we make this configurations on a kubernets runner created using gitlab.com – Coralyn

Hi @Joselow please help me on the same solution, please check the link once https://mcmap.net/q/455348/-gitlab-ci-getting-pending-status-after-push-on-the-gitlab/3946958 – Loiseloiter

sure this is nice, but if you are just a user and the admin refuses to do this ? – Boaten

Answer 2

I tried adding this to my helm chart deployment under runner configuration and docker started working, use this if you deployed your runner in Kubernetes.

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:18.04"
        privileged = true
        [[runners.kubernetes.volumes.host_path]]
          name = "docker"
          mount_path = "/var/run/docker.sock"

Hedge

Answer 3

this worked for me. specifically the DOCKER_HOST variable was the final straw to get success. couldn't tell you why though, sorry!

# .gitlab-ci.yml

# github runs docker right out of the box
# gitlab... not so much apparently
# https://mcmap.net/q/588201/-why-gitlab-runner-throws-quot-is-the-docker-daemon-running-quot
services:
  - docker:dind

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""
  DOCKER_DRIVER: overlay2

Bruyn

Answer 4

Run Gitlab Runner (With Privileges)

It seems that even with the additional privileges provided to the Bitnami gitlab-runner container, it is still encountering permission issues when trying to access the Docker daemon socket.

To resolve this issue, you can try the following steps:

1. Check the ownership and permissions of the Docker socket file on your macOS machine. Open a terminal and run the following command:

   ls -l /var/run/docker.sock
   

   Ensure that the file is owned by the root user and the staff group. The permissions should allow read and write access for the owner and group.

2. Adjust the ownership and permissions of the Docker socket file if necessary. Run the following command in the terminal:

   sudo chown root:staff /var/run/docker.sock
   sudo chmod 660 /var/run/docker.sock
   

3. Restart the Bitnami gitlab-runner container with the updated ownership and permissions of the Docker socket. Run the following command:

   sudo docker run -it --user=root -v /var/run/docker.sock:/var/run/docker.sock -v /Users/Shared/gitlab-runner/config:/etc/gitlab-runner bitnami/gitlab-runner
   

If you still unable to run the previous command,,

Run this final command to ensure Gitlab-runner container is running with root privilege, allowing it to access the Docker Daemon Socket.

By specifying --user=root in the command, you ensure that the Bitnami gitlab-runner container runs with root privileges, which should allow it to access the Docker daemon socket.

With these steps, the Bitnami gitlab-runner container should be able to connect to the Docker daemon and execute Docker-related tasks without encountering permission denied errors.

sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v /Users/Shared/gitlab-runner/config:/home/gitlab-runner/.gitlab-runner bitnami/gitlab-runner

Allinclusive

Answer 5

Had to add socket as volume as well:

[[runners]]
  name = "Builder"
  url = "https://gitlab.xxx.eu"
  id = 0
  token = "XXX"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["gitlab-shared-cache:/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

Middleaged

that was my case, ohhh, thanks! <3 – Vorster

Answer 6

The way 2 of @grapes's response didn't worked for me because i have several different runners. The "docker" group do not have the same ID on each of them. This mean that user of the container sometimes didn't has right to access to docker socket mounted on it.

You can easily check if you are in same case by running this on your host os:

cat /etc/group | grep docker && docker run -ti <your-docker-image-that-run-job> cat /etc/group | grep docker

For me it return:

docker:x:999:<host-username>,gitlab-runner
docker:x:998:<container-username>

If both id are different you probably has the same issue.

I had to add the option group_add = ["999"] in section [runners.docker] of my runner configuration (999 is docker's group id for this runner). This will automatically add container's user into docker's group for the runner.

Gitlab runner's documentation here.

  [runners.docker]
    tls_verify = false
    image = "ruby:2.7"
    privileged = false    
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]    
    shm_size = 0
    group_add = ["999"]

Jubbulpore