GitLab-CI: Cannot link to a non running container

Question

I've tried to get my setup work with gitlab-ci. I have a simple gitlab-ci.yml file

build_ubuntu:
  image: ubuntu:14.04
  services:
    - rikorose/gcc-cmake:gcc-5
  stage: build
  script:
    - apt-get update
    - apt-get install -y python3 build-essential curl
    - cmake --version
  tags:
    - linux

I want to get a ubuntu 14.04 LTS with gcc and cmake (apt-get version is to old) installed. If i use it locally (via docker --link command) everything works, but when the gitlab-ci-runner will process it i get the following waring (which is in my case an error)

Running with gitlab-ci-multi-runner 9.2.0 (adfc387)
on xubuntuci1 (19c6d3ce)
Using Docker executor with image ubuntu:14.04 ...
Starting service rikorose/gcc-cmake:gcc-5 ...
Pulling docker image rikorose/gcc-cmake:gcc-5 ...
Using docker image rikorose/gcc-cmake:gcc-5 
ID=sha256:ef2ac00b36e638897a2046c954e89ea953cfd5c257bf60103e32880e88299608 
for rikorose/gcc-cmake service...
Waiting for services to be up and running...

*** WARNING: Service runner-19c6d3ce-project-54-concurrent-0-rikorose__gcc-
cmake probably didn't start properly.

Error response from daemon: Cannot link to a non running container: /runner-
19c6d3ce-project-54-concurrent-0-rikorose__gcc-cmake AS /runner-19c6d3ce-
project-54-concurrent-0-rikorose__gcc-cmake-wait-for-service/runner-
19c6d3ce-project-54-concurrent-0-rikorose__gcc-cmake

Does anybody know how i can fix this?

Thanks in advance Tonka

Conchitaconchobar

How do you run rikorose/gcc-cmake:gcc-5 locally? – Aubigny

docker run -itd rikorose/gcc-cmake:gcc-5 --link=myrunningubuntucontainer – Conchitaconchobar

Do you use shared runners or have you set up your own runner? – Triphammer

It's my own hosted shared runner – Conchitaconchobar

Answer 1

You must start the gitlab-runner container with

--privileged true

but that is not enough. Any runner containers that are spun up by gitlab after registering need to be privileged too. So you need to mount the gitlab-runner

docker exec -it runner /bin/bash
nano /etc/gitlab-runner/config.toml

and change privileged flag from false into true:

privileged = true

That will solve the problem!

note: you can also mount the config.toml as a volume on the container then you won't have to log into the container to change privileged to true because you can preconfigure the container before running it.

Indicatory

Exactly this fixed it for me, many thanks! As far as I can see it's not mentioned in the official instructions.... – Rasbora

I already start container in privileged mode. This does not helped me. – Agogue

Starting the container in privilege mode is not enough. Did you log in to the container and changed the privileged flag in the config.toml? – Indicatory

@Indicatory I don't get it. Why I need to loging to container, this is not normal. – Rutledge

@holms, it is if you think about how gitlab runners work. The gitlab ci-cd is a subsystem that uses the gitlab-runner container you defined to start copies of itself to host multiple concurrent build stages and processes. They all need to be privileged. So the first flag is for the first container, the second is for the all the containers that are spun up by gitlab. – Indicatory

Making runners privileged is an awful solution to any problem that should not require these privileges. This introduces a huge security hole in your setup. What may be happening is that your service runs and then fails. I encountered this problem because my job caused the service to crash, so it was unable to attach to it afterwards, as it was no longer running. – Diatom

@Diatom it maybe a terrible solution but it's the one recommended by gitlab, somewhere, and moreover it works. If you want to make your runners more secure, point them to your ssh keys and turn on ssh, there's nothing more secure then that. – Indicatory

Answer 2

Though my CI configuration is different and it is serving a different purpose, I came across a similar issue basically resulting in the same error message referring to the attempt to link some non-running container.

Related runner has been working privileged for years. However, the service used in the particular project's .gitlab-ci.yml file was pretty outdated. It was including this definition:

  services:
    - docker:18.09-dind

Docker on host was already at v23.x, so changing that definition accordingly fixed it for me:

  services:
    - docker:23-dind

Namesake

In this case, maybe it is better to use docker:dind or docker:stable-dind without hardcoded version. See all available tags at hub.docker.com/_/docker/tags?name=dind – Nassir

Answer 3

In my case, I had to add

variables:
  DOCKER_TLS_CERTDIR: ""

Fidellas

On a side-note: This usually is done to force dind to start in non-TLS mode, other users use command: ["--tls=false"] on the service to achieve this. For me, the tls=false didnt worked, but specifying the variable did. Its also possible to use dind with TLS, but it requires sharing a volume with the certs between the task-container and the service. I believe TLS-as-default (=port 2736, not 2735) came after dind-v19. – Staffman

Answer 4

In my case with self-hosted on cloud VM GitLab runner this issue was caused by specifying wait_for_services_timeout = -1 in this config

[[runners]]
  [runners.docker]
    tls_verify = false
    privileged = false
    wait_for_services_timeout = -1

Unteach