gitlab: Runner is offline, last contact was about some hours ago

Question

recently my runners have been stopped and I don't know why?

I've just upgraded nodejs on the server and it did happen.

after this problem, I've tried to update gitlab to the latest version and check the runner status but the problem still persists and in the title of grey icon shows:

Runner is offline, the last contact was about 22 hours ago.

What should I do?

and when I try to Retry stuck jobs, see this error:

This job is stuck, because you don't have any active runners online with any of these tags assigned to them: 'my label'.

Any Help is appreciated!

Chaim

Answer 1

I don't why but I found the gitlab-runner was completely out of service. I've tried to fix it. at the first step, I've updated the gitlab-runner bit it didn't work. after that I decided to reinstall the gitlab-runner so I've met this steps:

1. removing the gitlab-runner user with this command line: sudo deluser --remove-home gitlab-runner
2. following the manual installation and configuration on here.

and it worked.

Chaim

Yes. For me worked too. But only if install from package and not by "sudo apt-get install gitlab-runner" command. – Rainband

Answer 2

To me, the following solved the problem:

gitlab-runner restart

Where gitlab-runner is a symlink to gitlab-ci-multi-runner:

GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI, the open-source continuous integration service included with GitLab that coordinates the jobs.

Gravitative

I tried to restart several times but it did not work – Neibart

This didn't worked for me. Instead, sudo gitlab-runner verify restarted all the runners – Cocker

Answer 3

This worked for me:

sudo gitlab-runner verify

since gitlab-runner restart didn't actually restarted runners

Cocker

This fixed it for me. I had registered multiple runners and removed them from the Gitlab UI. This cleaned up the stale runners. – Southeastward

This helped me realize I'd goofed in my config.toml file. – Persson

How do you do that through the UI though? I don't think there's a command line in there anywhere that I can access. – Babel

This helped me see that it was a certificate issue – Windgall

Answer 4

If you runner is run under systemd

journalctl -u gitlab-runner.service

may prove enlightening. It did in my case.

gitlab-runner[18449]: FATAL: chdir /home/gitlab-runner: no such file or directory

Jacobsen

Answer 5

Sharing my experience for future readers.

In my case, the internet was not working on the machine in which gitlab-runner was installed.

I have fixed the internet issue on my gitlab-runner's machine and everything works as expected.

Thanks!

Erelia

I don't have internet access on that machine as well, but I can't fix that. Why does it need internet connection anyway? – Lingulate

Answer 6

Check gitlab-runner logs with journalctl -u gitlab-runner -f

For me the problem was:

gitlab-runner[2023]: FATAL: Service run failed                           error=chdir /var/lib/gitlab-runner: no such file or directory

so created the folder and restarted gitlab-runner

sudo mkdir /var/lib/gitlab-runner
sudo gitlab-runner restart

probably something to do with installation not run by root user

Diaconal

Answer 7

For me the problem was, that the URL of our self-hosted GitLab changed. I had to change the URL of the runners manually:

sudo vi /etc/gitlab-runner/config.toml

After restarting, I could verify that all runners were back online:

sudo gitlab-runner restart
sudo gitlab-runner verify

Mignonne

Answer 8

after restart gitlab-runner , you can use this for new register runner :

gitlab-runner restart

gitlab-runner register --name my-runner --url http://gitlab.example.com --registration-token my-registration-token

Slavophile

Answer 9

In my case, after upgrading from an old gitlab-runner (9.5.1 -> 11.11.4) I couldn't get my runner to work. Even after removing it and registering a new one. I noticed that my config.toml file didn't have the token that matches the one in the Runner CI/CD Settings page for my new runner.

I did the following to get it to work:

1. Remove the /etc/gitlab-runner/config.toml file (perhaps unnecessary)

2. Click on Reset runners registration token in Gitlab under project's CI/CD Settings -> Runner page while registering a new runner (gitlab-runner register)

3. Continue with runner registration as usual.

After that, my runner was picked up and worked as usual.

Damien

Answer 10

I had the same error but using gitlab runner on GKE. To fix it I just deleted the gitlab-runner pod. A new pod was initialized and the connection was up again.

Emie

Answer 11

Sorry for resurrecting this old thread.

I'm currently working on making a gitlab runner that works locally. Still in the early phases, but eventually it will become very relevant. It doesn't seem like gitlab want/have time to make this, so here you go. https://github.com/firecow/gitlab-runner-local

Metalloid

Answer 12

For gitlab 15.7.1 and 15.8.0, this is what I did to resolve disconnecting runners symptoms:

• make sure home directory exists, create if it doesn't:

  sudo mkhomedir_helper gitlab-runner
  

• make sure --working-directory is set to gitlab-runner home in /etc/systemd/system/gitlab-runner.service

• running stop told me to run daemon-reload and then restarted the service:

  sudo service gitlab-runner stop
  sudo systemctl daemon-reload
  sudo service gitlab-runner stop
  sudo service gitlab-runner start
  sudo service gitlab-runner status
  

Rosalindrosalinda

Answer 13

for me its worked just run below command:

sudo gitlab-runner run

Companionate

Please use code snippet for highlight – Danielledaniels