Gitlab Runner with Docker and shell error — Permission denied

Question

Installed a brand new Gitlab CE 13.9.1 on a Ubuntu Server 20.04.2.0. This is the pipeline

image: node:latest

before_script:
  - apt-get update -qq

stages:
  - install

install:
  stage: install
  script:
    - npm install --verbose

To run it I configure my Gitlab Runner using the same procedure as in my previous Gitlab CE 12:

I pull last Gitlab runner image:

docker pull gitlab/gitlab-runner:latest

---

First try:

Start GitLab Runner container mounting on local volume

docker run -d \
--name gitlab-runner \
--restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

And register runner

docker run --rm -t -i \
-v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

When registering runner, for executor I pick shell

Finally, when I push to Gitlab, on the pipeline, I see this error:

$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1

---

Second try:

Start GitLab Runner container mounting on Docker volume

1. Create volume

docker volume create gitlab-runner-config

2. Start GitLab Runner container

docker run -d \
--name gitlab-runner \
--restart always \
-v gitlab-runner-config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

3. Register runner (picking shell again as executor)

docker run \
--rm -t -i \
-v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner register

Same results.

$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1

---

Third try:

Granting permissions to gitlab-runner

I ended up reading In gitlab CI the gitlab runner choose wrong executor and https://docs.gitlab.com/runner/executors/shell.html#running-as-unprivileged-user, which states these solutions:

1. move to docker
2. grant user gitlab-runner the permissions he needs to run specified commands. gitlab-runner may run apt-get without sudo, also he will need perms for npm install and npm run.
3. grant sudo nopasswd to user gitlab-runner. Add gitlab-runner ALL=(ALL) NOPASSWD: ALL (or similar) to /etc/sudoers on the machine gitlab-runner is installed and change the lines apt-get update to sudo apt-get update, which will execute them as privileged user (root).

1. I need to use shell
2. I already did that with sudo usermod -aG docker gitlab-runner
3. Tried as well with sudo nano /etc/sudoers, adding gitlab-runner ALL=(ALL) NOPASSWD: ALL, and using sudo apt-get update -qq in the pipeline, which results in bash: line 106: sudo: command not found

I'm pretty lost here now. Any idea will be welcome.

Airspeed

Answer 1

IMHO, using shell executor on a Docker runner with already mounted Docker socket on it is not a good idea. You'd better use docker executor, which will take care of everything and probably is how it's supposed to be run.

Edit

Alternatively, you can use a customized Docker image to allow using the shell executor with root permissions. First, you'll need to create a Dockerfile:

FROM gitlab/gitlab-runner:latest
# Change user to root
USER root

Then, you'll have to build the image (here, I tagged it as custom-gitlab-runner):

$ docker build -t custom-gitlab-runner .

Finally, you'll need to use this image:

docker run -d \
  --name gitlab-runner \
  --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  custom-gitlab-runner:latest

Linstock

As stated in the question shell is a requirement. – Airspeed

This is not stated as a requirement. Furthermore, you are configuring everything as if you are going to use the docker executor (using Docker In Docker). – Linstock

Please refer to the title of the question. I'm aware about the configuration of the pipeline as if I were going to use it with docker executor; the interesting thing here is that I had it running this pipeline with Gitlab Runner 12 with shell executor. Should work this way, I'm pretty sure I'm doing something wrong. – Airspeed

Please refer to the title of the question It's not stating it as a requirement, either. The posted solution is -at least- an adequate one, as it fixes the error (just change shell to docker and everything will work). Downvote is not fair, or edit your question and ensure shell is stated as a requirement (which will be difficult to understand, but will be your specific question). – Linstock

You are in reason, just edit the answer in anyway so I can unvote it – Airspeed

Added alternate solution – Linstock

Using your solution,seems to work, tons of thanks. – Airspeed

Note: The alternate solution may have some security implications and other kind of errors (for example, think about side effects). – Linstock

tons of thanks You're welcome :) Please, also mark the answer as the correct one. – Linstock

«The alternate solution may have some security implications»: true, going with docker executor is definitely the way to go – Airspeed

Answer 2

I had a similar issue trying to use locally installed gitlab-runner on ubuntu with a shell executor (I had other issues using docker executor not being able to communicate between stages)

$ docker build -t myapp .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=myapp&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied
ERROR: Job failed: exit status 1

I then printed what user was running the docker command within the gitlab-ci.yml file, which was gitlab-runner

...

build:
  script:
    - echo $USER
    - docker build -t myapp .
...

I then added gitlab-runner to the docker group using

sudo usermod -aG docker gitlab-runner 

which fixed my issue. No more docker permission errors.

Sweetbread