How gitlab runner concurrency works?

Question

Q1: Whats the difference between

concurrent = 3

[[runners]]
  ..
  executor = "shell"

and

concurrent = 3

[[runners]]
  ...
  executor = "shell"

[[runners]]
  ...
  executor = "shell"

[[runners]]
  ...
  executor = "shell"

Q2: Does it makes sense, to...

have 3 executors (workers) of same type on a single runner with global concurrent = 3? Or can single executor with global concurrent = 3 do multiple jobs in parallel safely?

Q3: How they're related...

runners.limit with runners.request_concurrency and concurrent

Thanks

Psychodiagnostics

Answer 1

Gitlab's documentation on runners describes them as:

(...) isolated (virtual) machines that pick up jobs through the coordinator API of GitLab CI

Therefore, each runner is an isolated process responsible for picking up requests for job executions and for dealing with them according to pre-defined configurations. As an isolated process, each runner have the capability of creating 'sub-processes' (also called machines) in order to run jobs.

When you define in your config.toml a [[runner]] section, you're configuring a runner and setting how it should deal with job execution requests. In your questions, you mentioned two of those "how to deal with job execution request"' settings:

1. limit: "Limit how many jobs can be handled concurrently". In other words, how many 'sub-processes' can be created by a runner in order to execute jobs simultaneously;
2. request_concurrency: "Limit number of concurrent requests for new jobs from GitLab". In other words, how many job execution requests can a runner take from GitLab CI job queue simultaneously.

Also, there are some settings that apply to a machine globally. In your question you mentioned one of them:

1. concurrent: "Limit how many jobs globally can be run concurrently. This is the most upper limit of number of jobs using all defined runners". In other words, it limits the maximum amount of 'sub-processes' that can run jobs simultaneously.

Thus, keeping in mind the difference between a runner its sub-processes and also the difference between specific runner settings and global machine settings:

Q1:

The difference is that in your 1st example you have one runner and in your 2nd example you have three runners. It's worth mentioning that in both examples your machine would only allow running 3 jobs simultaneously.

Q2:

Not only a single runner can run multiple jobs concurrently safely but also is possible to control how many jobs you want it to handle (using the aforementioned limit setting).

Also, there is no problem to have similar runners running in the same machine. How you're going to define your runner's configurations is up to you and your infrastructure capabilities.

Also, please notice that an executor only defines how to run your job. It isn't the only thing that defines a runner and it isn't a synonymous for "worker". The ones working are your runners and their sub-processes.

Q3:

To summarize: You can define one or many workers at the same machine. Each one is an isolated process. A runner's limit is how many sub-processes of a runner process can be created to run jobs concurrently. A runner's request_concurrency is how many requests can a runner handle from the Gitlab CI job queue. Finally, setting a value to concurrent will limit how many jobs can be executed at your machine at the same time in the one or more runners running in the machine.

References

For better understanding, I really recommend you read about Autoscaling algorithm and parameters.

Finally, I think you might find this question on how to run runners in parallel on the same server useful.

Road

Thanks for your answer. To verify if i understand it correctly: if I have concurrent=1 and 3 workers, then only 1 of them can take the job, while other 2 will be idle. on the other hand, when i have concurrent = 3, and 3 workers each with limit = 1, then single machine can run 3 jobs in parallel, but each worker can take only 1 job. Correct? – Psychodiagnostics

how can be concurrent to be set programmaticaly? – Ptosis

The difference between limit and request_concurrency is still unclear for me. – Gunderson

If only one runner defined on this machine, and concurrent=3, limit=3, what's the difference for request_concurrency=1 or 3? – Ginglymus

The effective limit on the number of concurrent jobs executed by a single runner, not accounting for the global concurrent limit, appears to be limit == 0 ? request_concurrency : min(limit, request_concurrency). Why in the world this configuration is so confusing and split into two different settings when one would obviously suffice, I cannot say. (default value for limit is 0 and request_concurrency is 1 so just ignore limit and set request_concurrency 🤮) – Register

@Gunderson I think this should be added to the answer. I don't know if there is any report, but at least on gitlab-runner 14.3.2 limit is used to work around a bug: if you set request_concurrency = 1, it actually will be ignored, I presume in preference of the global concurrent setting. In my tests the runner with request_concurrency = 1 will happily take 2 jobs simultaneously (which is usually not something you'd want, there's usually a reason for limiting the concurrency, such as access to a shared resource). To make it actually work you have to additionally set limit = 1. – Peddada

Answer 2

Re: limit vs request_concurrency

limit: Run up to limit jobs concurrently.
request_concurrency: Accept up to request_concurrency job requests. In other words, queue these jobs to run on this runner when less than limit jobs are running.

This runner will queue up to 4 jobs and run up to 2 of them concurrently.

[[runners]]
  limit = 2
  request_concurrency = 4

Source: https://www.howtogeek.com/devops/how-to-manage-gitlab-runner-concurrency-for-parallel-ci-jobs/#handling-request-concurrency

Vert