stable/prometheus-operator - adding persistent grafana dashboards

Question

I am trying to add a new dashboard to the below helm chart

https://github.com/helm/charts/tree/master/stable/prometheus-operator

The documentation is not very clear.

I have added a config map to the name space like the below -

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-grafana-dashboard
  namespace: monitoring
  labels:
     grafana_dashboard: "1"
data:
  etcd-dashboard.json: |-
{JSON}

According to the documentation, this should just be "picked" up and added, but its not. https://github.com/helm/charts/tree/master/stable/grafana#configuration

The sidecar option in my values.yaml looks like -

grafana:
  enabled: true

  ## Deploy default dashboards.
  ##
  defaultDashboardsEnabled: true

  adminPassword: password

  ingress:
    ## If true, Grafana Ingress will be created
    ##
    enabled: false

    ## Annotations for Grafana Ingress
    ##
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"

    ## Labels to be added to the Ingress
    ##
    labels: {}

    ## Hostnames.
    ## Must be provided if Ingress is enable.
    ##
    # hosts:
    #   - grafana.domain.com
    hosts: []

    ## Path for grafana ingress
    path: /

    ## TLS configuration for grafana Ingress
    ## Secret must be manually created in the namespace
    ##
    tls: []
    # - secretName: grafana-general-tls
    #   hosts:
    #   - grafana.example.com
  #dashboardsConfigMaps:
    #sidecarProvider: sample-grafana-dashboard
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard

I have also tried adding this to the value.yml

dashboardsConfigMaps:
   - sample-grafana-dashboard

Which, doesn't work.

Does anyone have any experience with adding your own dashboards to this helm chart as I really am at my wits end.

Coltin

please add your helm command – Traditional

helm install stable/prometheus-operator --name prometheus-operator -f values.yml --namespace monitoring – Coltin

is your sidecar block under grafana ? – Traditional

Yep, edited main post so you can see. according to the documentation, it should just pick up the config map if it is labeled with grafana_dashboard - doesnt seem to be happening. – Coltin

Ok, this has magically started working now – Coltin

Answer 1

To sum up: For sidecar you need only one option set to true - grafana.sidecar.dashboards.enabled

1. Install prometheus-operator witch sidecard enabled:

helm install stable/prometheus-operator --name prometheus-operator --set grafana.sidecar.dashboards.enabled=true --namespace monitoring

2. Add new dashboard, for example MongoDB_Overview:

wget https://raw.githubusercontent.com/percona/grafana-dashboards/master/dashboards/MongoDB_Overview.json
kubectl -n monitoring create cm grafana-mongodb-overview --from-file=MongoDB_Overview.json

3. Now the tricky part, you have to set a correct label for your configmap, by default grafana.sidecar.dashboards.label is set tografana_dashboard, so:

kubectl -n monitoring label cm grafana-mongodb-overview grafana_dashboard=mongodb-overview

Now you should find your newly added dashboard in grafana, moreover every confimap with label grafana_dashboard will be processed as dashboard.

The dashboard is persisted and safe, stored in configmap.

UPDATE:

January 2021:

Prometheus operator chart was migrated from stable repo to Prometheus Community Kubernetes Helm Charts and helm v3 was released so:

1. Create namespace:

kubectl create namespace monitoring

2. Install prometheus-operator from helm chart:

helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring

3. Add Mongodb dashboard as an example

wget https://raw.githubusercontent.com/percona/grafana-dashboards/master/dashboards/MongoDB_Overview.json
kubectl -n monitoring create cm grafana-mongodb-overview --from-file=MongoDB_Overview.json

4. Lastly, label the dashboard:

kubectl -n monitoring label cm grafana-mongodb-overview grafana_dashboard="1"

Intertidal

This should be the selected answer ^ – Macfadyn

Does the dashboard configmap have to reside in the same namespace (monitoring) in order for Grafana to pick it up? If so, is there a way Grafana can be configured to scan other namespaces for dashboard configmaps? – Rania

1. As of January 2021, grafana.sidecar.dashboards.enabled=true is activated by default. You can check it with helm show values prometheus-community/kube-prometheus-stack | less and searching for grafana 2. How long does it take for grafana sidecar to pick up the added ConfigMap? I waited for some minutes - the new dashboard did not appear. Then I killed the pod and my dashboard appeared. Are you supposed to manually restart the grafana pod to show new dashboards? – Quadruplet

For this to work, the label to add to the cm should be grafana_dashboard=1 instead of grafana_dashboard=mongodb-overview, as that is the default value the helm chart will be looking for unless you change it when installing the chart. – Endorsee

Answer 2

You have to:

• define you dashboard json as a configmap (as you have done, but see below for an easier way)
• define a provider: to tell where to load the dashboard
• map the two together

from values.yml:

  dashboardsConfigMaps:
    application: application
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: application
          orgId: 1
          folder: "Application Metrics"
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/application

Now the application config map should create files in this directory in the pod, and as has been discussed the sidecar should load them into an Application Metrics folder, seen in the GUI.

That probably answers your issue as written, but as long as your dashboards aren't too big using kustonmise mean you can have the json on disk without needing to include the json in another file thus:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# May choose to enable this if need to refer to configmaps outside of kustomize
generatorOptions:
  disableNameSuffixHash: true

namespace: monitoring

configMapGenerator:
- name: application
  files:
    - grafana-dashboards/application/api01.json
    - grafana-dashboards/application/api02.json

For completeness sake you can also load dashboards from url or from the Grafana site, although I don't believe mixing method in the same folder works.

So:

  dashboards:
    kafka:
      kafka01:
        url: https://raw.githubusercontent.com/kudobuilder/operators/master/repository/kafka/docs/latest/resources/grafana-dashboard.json
        folder: "KUDO Kafka"
        datasource: Prometheus
    nginx:
      nginx1:
        gnetId: 9614
        datasource: Prometheus
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: kafka
          orgId: 1
          folder: "KUDO Kafka"
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/kafka
        - name: nginx
          orgId: 1
          folder: Nginx
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/nginx

Creates two new folders containing a dashboard each, from external sources, or maybe you point this at your git repo you de-couple your dashboard commits from your deployment.

Warhorse

Does dashboardsConfigMaps: support wildcards? EG: dashboardsConfigMaps: application: *-application – Meliorate

Answer 3

If you do not change the settings in the helm chart. The default user/password for grafana is: user: admin password: prom-operator

Merv

Thanks for your suggestion, but the OP is looking for a way to statically add Dashboard via an Helm release. – Macfadyn