How can I disable concurrent jobs on the same gitlab runner with docker executor?

Question

I am using gitlab community edition 14.4.1 along with one single gitlab runner with version 14.4.0. Its configuration reads as follows:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "My runner"
  limit = 1
  url = "https://my-gitlab-instance.com"
  token = "my-gitlab-token"
  executor = "docker"
  [runners.custom_build_dir]
    enabled = true
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "gitlab/dind:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/builds:/builds:rw", "/cache"]
    shm_size = 1000000000

Note in particular the two options

concurrent = 1

and

[[runners]]
  limit = 1

Now, I have a pipeline where some stages have multiple jobs. From the above runner configuration, I expect that each job would run one after the other, sequentially. However, the jobs are run concurrently on the same runner, causing most of them to fail because of git locks.

Why is that so? How can I really disable concurrency within a runner? It seems like disabling concurrency is my only option. Indeed, I tried to investigate how to make it work in a concurrent way by e.g. defining in my .gitlab-ci.yml the GIT_CLONE_PATH like this

variables:
  GIT_CLONE_PATH: ${CI_BUILDS_DIR}/${CI_CONCURRENT_ID}/${CI_PROJECT_NAME}

but that doesn't work because the $CI_CONCURRENT_ID is not filled correctly by gitlab (it's always 0, whatever happens).

Increasing the number of runners is not a solution, because I can still observe the very same effect. Sometimes, multiple jobs would still run on the same runner at about the same time. It cannot be that there is no work-around, right? How can I solve this issue?

In the end, I want to use multiple runners, but of course with a distribution of max one job / runner otherwise my pipelines are not reliable (i.e. they can fail because of those annoying git locks errors). For that to work, I need concurrency within my runners to be disabled.

Currycomb

Your config looks correct. You should only have 1 job run at a time if you have only 1 runner. It sounds like you may have more than 1 runner unexpectedly registered and running. Check your registered runners in the UI to confirm there's only 1 registered runner. Double check your runner config file that it contains only one runner entirely and restart the runner to make sure it's using the latest configuration. echo $CI_RUNNER_ID in your job to confirm your jobs are using the same singular runner. – Glynisglynn

Well, they use the same singular runner, because most of the time one the jobs will crash because of the git locks error, because they all run concurrently on the same runner. When I look at these jobs on gitlab, I can clearly see that they have all run on the same runner (the runner's name is displayed on the right). – Currycomb

@LaurentMichel same issue here. Have you had any luck solving it since ? – Acanthaceous

No, no "luck". And that's the problem with gitlab. I have the impression, I need "luck" when I work with gitlab. I'm wondering how that software can reasonably be used as a serious CI/CD tool. For my little personal projects it's fine, but I would never use it professionally. – Currycomb

Same issue here, no luck. I'm sad that you have it the same way. It looks like I'll have to switch to another CI/CD tool bcs of this. – Eryn

@LaurentMichel did you ever solve this? Running into a similar issue. Having a hard time believing that this is unsupported behavior... – Alleged

Nope, I never solved it, I switched to TeamCity instead, I was sick of gitlab CI – Currycomb

Answer 1

If u are looking to run jobs in a stage one at a time u can use resource groups option.

job1:
 resource_group: team
 script:
   - echo "Hello"

job2:
 resource_group: team
 script:
   - echo "hai"

In this case job1 will run and only after it is completed the other job will run.

Forseti

I want to run one at a time on a given runner. If I have one single runner, I want to run all the jobs sequentially. If I have 2 runners, I want that each runner be executing jobs in parallel. In that case, I don't want the same runner to take more than 1 job at a time. – Currycomb

Answer 2

Check that you don't have two instances of gitlab_runner running.

You should see something like the following when you look at running processes. If you see two entries for gitlab-runner you have an issue.

# example of running processes on *nix using ps
$ ps -ef | grep gitlab
  502   852     1   0 12:52pm ??         0:00.08 /path/to/gitlab-runner run --syslog
  502  1926   389   0 12:53pm ttys000    0:00.00 grep gitlab

NOTE: if there are also short-lived gitlab-runner verify commands listed, those are ok.

See:

• https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29158
• https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29202

Trumaine