How can I give grafana user appropriate permission so that it can start successfully?

Question

env:

kubernetes provider: gke
kubernetes version: v1.13.12-gke.25
grafana version: 6.6.2 (official image)

grafana deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:6.6.2
        ports:
        - name: grafana
          containerPort: 3000
        # securityContext:
        #     runAsUser: 104
        #     allowPrivilegeEscalation: true
        resources:
          limits:
            memory: "1Gi"
            cpu: "500m"
          requests: 
            memory: "500Mi"
            cpu: "100m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
              claimName: grafana-pvc

Problem

when I deployed this grafana dashboard first time, its working fine. after sometime I restarted the pod to check whether volume mount is working or not. after restarting, I getting below error.

mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later

what I understand from this error, user could create these files. How can I give this user appropriate permission to start grafana successfully?

Nearsighted

Answer 1

I recreated your deployment with appropriate PVC and noticed that grafana pod was failing.

Output of command: $ kubectl get pods -n monitoring

NAME READY STATUS RESTARTS AGE
grafana-6466cd95b5-4g95f 0/1 Error  2  65s

Further investigation pointed the same errors as yours:

mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later

This error showed on first creation of a pod and the deployment. There was no need to recreate any pods.

What I did to make it work was to edit your deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      securityContext:
          runAsUser: 472
          fsGroup: 472
      containers:
      - name: grafana
        image: grafana/grafana:6.6.2
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "500m"
          requests:
            memory: "500Mi"
            cpu: "100m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
              claimName: grafana-pvc

Please take a specific look on part:

      securityContext:
          runAsUser: 472
          fsGroup: 472

It is a setting described in official documentation: Kubernetes.io: set the security context for a pod

Please take a look on this Github issue which is similar to yours and pointed me to solution that allowed pod to spawn correctly:

• https://github.com/grafana/grafana-docker/issues/167

Grafana had some major updates starting from version 5.1. Please take a look: Grafana.com: Docs: Migrate to v5.1 or later

Please let me know if this helps.

Temikatemp

This answer helped a lot, but didnt quite work for me. I also had to add runAsUser: 0 to the securityContext parameters to get going. Thanks – Crossruff

Answer 2

On v8.0, I do that setting runAsUser: 0. It works.

---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
      - name: grafana-tcp
        port: 3000
        protocol: TCP
        targetPort: 3000
  selector:
      project: grafana
  type: LoadBalancer
status:
    loadBalancer: {}

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
      project: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
        project: grafana
  strategy:
      type: RollingUpdate
  template:
    metadata:
        labels:
            project: grafana
        name: grafana
    spec:
        securityContext:
          runAsUser: 0
        containers:
          - image: grafana/grafana
            name: grafana
            ports:
              - containerPort: 3000
                protocol: TCP
            resources: {}
            volumeMounts:
              - mountPath: /var/lib/grafana
                name: grafana-volume
        volumes:
          - name: grafana-volume
            hostPath:
              # directory location on host
              path: /opt/grafana
              # this field is optional
              type: DirectoryOrCreate
        restartPolicy: Always
status: {}

Incredible

This is the only solution that worked for me on 9.* – Schott