Not able to execute GitLab Runner in Kubernetes cluster: cannot create resource "secrets" in API group "" in the namespace "gitlab"

Question

Currently I'm facing the issue:

ERROR: Job failed (system failure): 
prepare environment: 
setting up credentials: 
secrets is forbidden: 
User "system:serviceaccount:default:gitlab-runner" cannot create
resource "secrets" in API group "" in the namespace "gitlab" 

after following the official documentation on how to integrate the GitLab Runner.

I'm using the following runner-chart-values.yaml:

# The GitLab Server URL (with protocol) that want to register the runner against
# ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
#
gitlabUrl: http://example.domain/

# The Registration Token for adding new runners to the GitLab Server. This must
# be retrieved from your GitLab instance.
# ref: https://docs.gitlab.com/ce/ci/runners/README.html
#
runnerRegistrationToken: "<token>"

# For RBAC support:
rbac:
    create: true
    rules:
      - apiGroups: ["*"]

# Run all containers with the privileged flag enabled
# This will allow the docker:dind image to run if you need to run Docker
# commands. Please read the docs before turning this on:
# ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind
runners:
    privileged: true

Any clues what's going on?

Many thanks!

Cohdwell

Answer 1

Looks like there is namespace mismatch however you can try this below option

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "get", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]

make sure you are creating the service account of Role to proper namespace.

Command to create Role binding

kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runner --serviceaccount=gitlab-runner:default

here is nice documentation : https://medium.com/@ruben.laguna/installing-a-gitlab-runner-on-kubernetes-ac386c924bc8

Electrojet

Many thanks for your information. Did everything according to the medium article, but got the problem that the runner is created within the default namespace, even a different namespace got specified gitlab-runner – Cohdwell

Answer 2

For me adding all necessary roles was the only solution that actually helped.

Here the corresponding runner-chart-values.yaml file:

## GitLab Runner Image
gitlabUrl: http://example.domain/
runnerRegistrationToken: "<token>"

rbac:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["list", "get", "watch", "create", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]
    - apiGroups: [""]
      resources: ["pods/attach"]
      verbs: ["list", "get", "create", "delete", "update"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["list", "get", "create", "delete", "update"]      
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["list", "get", "create", "delete", "update"]      

runners:
  privileged: true

Themselves

Answer 3

Here is a complete solution using Helm, I copied the rights proposed by Richard in this answer.

Using the followin template (gitlab-rbac/templates) we could patch a given namespace using:

helm upgrade -i gitlab-rbac-name ./gitlab-rbac \
-n your-namespace-here --create-namespace

Once installed, you can check your current rights with:

kubectl auth can-i create secrets --as=system:serviceaccount:gitlab:default \
-n your-namespace-here

The template gitlab-rbac/templates/rbac.yaml contains the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Release.Namespace }}-admin
  namespace: {{ .Release.Namespace }}
rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["list", "get", "watch", "create", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]
    - apiGroups: [""]
      resources: ["pods/attach"]
      verbs: ["list", "get", "create", "delete", "update"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["list", "get", "create", "delete", "update"]      
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["list", "get", "create", "delete", "update"]  
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-runner-{{ .Release.Namespace }}-admin
  namespace: {{ .Release.Namespace }}
subjects:
  - kind: ServiceAccount
    name: default
    namespace: gitlab
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {{ .Release.Namespace }}-admin

Note that you might need a lot more rights for your runners, you will probably need to update the rules depending on your pipeline. For example if you allow your template to create namespaces, you will need to add a cluster wide role for this. This means adding the following in the template file:

# ... Role and RoleBinding templates
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-admin
rules:
  - apiGroups: ['']
    resources: ['namespaces']
    verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-runner-namespace-admin
subjects:
  - kind: ServiceAccount
    name: default
    namespace: gitlab
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: namespace-admin

The value (gitlab-rbac/values.yaml) file is empty in this example as we only use the namespace argument.

Waldenses

Are you sure about ClusterRoleBinding ? I didn't test yet but it seems OK with runners:privileged:true docs.gitlab.com/runner/install/… – Ametropia

I am using Role and RoleBinding and it works well. – Skees

Answer 4

I also got the same error. So I have used this method. It solved my error.

helm install --namespace <NAMESPACE> gitlab-runner -f values.yml --set rbac.create=true gitlab/gitlab-runner

Ravo

Answer 5

Extending Harsh's answer: Please make sure that you're working under active 'gitlab-runner' namespace or using the key --namespace=gitlab-runner. To switch between active namespaces, please use the following command:

kubens gitlab-runner

So you don't need to use --namespace=gitlab-runner everytime.

JFYI, I've done that steps from the article on my k8s cluster and it works fine for me.

Dominick

Answer 6

In addition to the other answers. Here is a link to the official documentation which lists what permissions are needed depending on the strategy used:

Link to documentation: https://docs.gitlab.com/runner/executors/kubernetes.html

Posting

Answer 7

You have to enable RBAC support. Or disable RBAC on your Kubernetes installation.

There is a section in the official documentation addressing this error: https://docs.gitlab.com/runner/install/kubernetes.html#error-job-failed-system-failure-secrets-is-forbidden.

Kamenskuralski