Gitlab: There has been a runner system failure

Question

I use Gitlab for doing Continuous Integration and Development and all of a sudden I get this error message "There has been a runner system failure, please try again"

There's no real error message or error code.

I've tried restarting the gitlab runner, using gitlab-runner restart, I've done a reboot of the server its running on but I keep getting this error message on Gitlab whenever I push a code change.

Tania

Answer 1

After a couple of hours, I realized the issue is that the server that Gitlab Runner is running on has no space left.

I logged into the server in question. Looked at the Gitlab log file using the following command: journalctl -u gitlab-runner

And it showed me the following logs:

May 21 08:20:41 gitlab-runner[18936]: Checking for jobs... received                       job=178911 repo_url=https://.......git runner=f540b942
May 21 08:20:41 gitlab-runner-01 gitlab-runner[18936]: WARNING: Failed to process runner                   builds=0 error=open /tmp/trace543210445: no space left on device executor=docker runner=f540b942

To fix this issue I ran the docker container prune command which clears out stopped containers.

Alternatively you could use the docker system prune command which would remove all unused objects.

See https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/ for more information about those docker commands.

Afterwards, I no longer got the error on Gitlab when pushing changes.

Tania

Just to add a small information that may be valuable for others because it took me some research and time, i had the same problem, it was also related to disk space being full, in my case the mysql volume was huge compared to the data it holds. I had to purge the binlog older than 3 days in order for it to work. Hope this helps someone facing the same issue ! – Albritton

Answer 2

In my case I got this error

WARNING: after_script failed, but job will continue unaffected: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/53675a50d5dd838707b97d7ab37f3ace6bcf80f2ac9d80171f7fe9f996bb7d8d/merged: invalid argument (exec.go:78:0s)  job=22255 project=7 runner=h8gbgAAh
Mar 25 09:13:43 <gitlab-runner-name> gitlab-runner[488]: WARNING: Error while executing file based variables removal script  error=Error response from daemon: mkdir /var/lib/docker/overlay2/3b5cacb1bf37d1bb771015f530df17d01a58a1e72e03e89504f10549123aac06/merged: read-only file system (exec.go:78:0s) job=22255 project=7 runner=h8gbgAAh
Mar 25 09:13:43 <gitlab-runner-name>gitlab-runner[488]: WARNING: Job failed: exit code 1

systemctl status docker.service showed that the service failed
sudo systemctl restart docker gave "Job for docker.service failed because the control process exited with error code"
sudo dockerd --debug as suggested, but not sure if it helped
=> sudo reboot brought everything back on tracks.

From Docker: How to fix "Job for docker.service failed because the control process exited with error code"

Essonite