How to run multiple gitlab-ci-runners?

Question

Well, very basic question but I just didn't find (good) documentation; here we go:

• Can I setup/run several runners on the same server?
• What means runner? Is this the (omnibus) installation or the processes I start with e.g. /opt/gitlab-runner/bin/runner?
• Regarding /opt/gitlab-runner/bin/runner: does it make sense to start several such processes? (They don't show up individually under the 'Runners' tab in the gitlab-ci webview)

Thanks.

Decoy

If you're using docker there's no need to run multiple runners on a server. Simply up the concurrent setting in the config. – Phebe

Answer 1

For anyone else trying to run multiple gitlab-runners on windows:

• Download/Copy the gitlabrunner.exe to two different locations.
• Register them in their seperate folders.
• Install them and provide different names to the -n flag e.g:

gitlab-runner-windows-amd64.exe install -n dockerbuilder

• Start them using the same names with the -n flag.

Sipe

This installs the Windows service under the exact name one provides. To see all the runners in Task Manager in one place one might prefix all names with gitlab-runner- – Idell

Answer 2

1. Yes you can setup multiple runners. See https://docs.gitlab.com/runner/ for more information
2. You usually setup runners on a different box from your main gitlab instance. At least that is what I do. Omnibus GITLAB refers to the main system where your repos are maintained and accessible through the web UI. Runners are isolated processes (workers) that wait for new commits to be pushed and then execute the build.
3. A good practice is to have multiple runners setup, dedicated to a particular technology. E.g. a dedicated runner for JAVA, NodeJS, Python, Ruby or PHP builds. If no runners show up in the UI, they are not registered correctly with your instance. See the documentation in #1

Collett

Answer 3

You can run multiple instances, but since gitlab 12.2 with important caveat: https://docs.gitlab.com/runner/faq/README.html#why-cant-i-run-more-than-one-instance-of-runner

Why can’t I run more than one instance of Runner?

You can, but not sharing the same config.toml file.

Running multiple instances of Runner using the same config file can cause unexpected and hard-to-debug behavior. In GitLab Runner 12.2, only a single instance of Runner can use a specific config.toml file at one time

Celebrity

Answer 4

First, understand why you want multiple runners. If you just want multiple jobs to run in parallel, the builtin executors will respect the concurrent configuration and allow execution of multiple jobs in parallel. Generally, you only need one runner process and one config file per host. This process can handle multiple runner configurations at the same time.

If you want multiple runners with different configurations (e.g., have a shell executor runner and a docker executor runner side-by-side), you can add multiple runner sections to the config file; just add another [[runner]] section to the config.toml!. If you call the register command multiple times, it will do this for you. Each runner will show up as a separate registered runner in the GitLab UI, but you only need one config and one gitlab-runner process to manage multiple registered runners in concert.

For example, a config with two separately configured executors you would run gitlab-runner register twice with different arguments for a different resulting configuration.

Start with a config file with just the global elements or other non-runner sections defined. Something like this:

concurrent = 10
check_interval = 0
# ... etc...

[session_server]
  session_timeout = 1800

Then register the runners you want with configuration keys passed by command line (or you can edit configuration in config.toml after registering each runner):

# register the first runner
gitlab-runner register \
  --non-interactive \
  --url "https://${HOSTNAME}/" \
  --registration-token "${REGISTRATION_TOKEN}" \
  --executor "docker" \
  --tag-list "docker" \
  --run-untagged="true" \
  --config="/etc/gitlab-runner/config.toml"
  # ... etc

# register the second runner
gitlab-runner register \
  --non-interactive \
  --url "https://${HOSTNAME}/" \
  --registration-token "${REGISTRATION_TOKEN}" \
  --executor "shell" \
  --tag-list "shell" \
  --limit="2" \
  --config="/etc/gitlab-runner/config.toml"
  # ... etc

Notably --tag-list must be provided at the time of registration. After registration, this can only be edited in the GitLab UI or via the API.

After running the register commands, this should result in your config.toml being edited with two [[runners]] sections. It would look something like this:

concurrent = 10  # global/total concurrency
check_interval = 0
# ... etc...

[session_server]
  session_timeout = 1800

# Specify a docker executor
[[runners]]
  (...)  # specific runner config, token, etc.
  executor = "docker"
  [runners.docker]
    (...)  # docker executor configuration...


# specify a shell executor
[[runners]]
  (...)  # specific runner config, token, etc.
  executor = "shell"
  limit = 2  # concurrency limit for THIS executor

When you run gitlab-runner start you should see two runners online in the GitLab UI

Colorist

Answer 5

I had a setup, where I needed exactly one runner for two distinct jobs. When you define concurrent, it sets concurrency for all runners in given machine.

If you have two runners (with different labels or setup) and concurrent=2, it means both may run 2 jobs in parallel. If concurrent=1, only one runner will run one job at a time.

Found solution here: https://forum.gitlab.com/t/run-runners-in-parallel-on-the-same-server/9213/2 concurrent=2 and for each runner add limit=1 to have max 2 jobs on the server and limit each runner with exactly one job.

Yeta

Answer 6

Since the link provided by Mateuszl doesn't exist any more, and I am not able to comment because I am a relatively new user, I'm sharing the new link here:

https://docs.gitlab.com/runner/faq/#why-cant-i-run-more-than-one-instance-of-gitlab-runner

You can run multiple runners but with this point in mind: In GitLab Runner 12.2, only a single instance of GitLab Runner can use a specific config.toml file at one time because of concurrency issues, and the issue linked has been closed with the accepted solution as running multiple runners pointing to different config toml files:

https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3688

https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4407

Confirmed