Formatting Slack Notification from Grafana

Question

Getting Alerts in slack from Grafana. But it's not in a good format. Is there any way to customize the notification in slack from Grafana?. I am using Node exporter to export the metrics. The dashboard is downloaded from https://grafana.com/grafana/dashboards/1860

How to customize the notification like:

Heading: ALERT CPU LOAD ON SERVER

Instance IP: 192.x.x.x.x Value: 90% of CPU used Priority: High

Sample output in slack is attached.

Palenque

I want this too - need a way to extract values from the Value: field. The templating tutorials I've tried don't address this. – Benne

I'm struggling as well, but this was a little helpful: youtube.com/… – Vostok

Answer 1

So i managed to work it out nicely according to my needs with the help of this guide :

https://community.grafana.com/t/how-to-use-alert-message-templates-in-grafana/67537/3

I will never the less post the config here as well:

1. Grafana / Contact points / Message template - Create new template called 'myalert'

{{ define "myalert" }}
  [{{.Status}}] {{ .Labels.alertname }} 
  {{ .Annotations.AlertValues }}
{{ end }}

2. Grafana / Contact points - Message template - Create another template for the message called 'mymessage'

{{ define "mymessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ len .Alerts.Firing }} firing:
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ len .Alerts.Resolved }} resolved:
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
  {{ end }}
{{ end }}

3. Grafana / Contact points / Contact points / Slack / Optional Slack settings / Text body - Add the following:

Alert summary:
{{ template "mymessage" . }}

4. Grafana / Some-Alert / Summary and Annoitations - add a field called 'AlertValues' with the following content:

Alert summary:
{{ with $values }}
{{ range $k, $v := . }}
   Location: {{$v.Labels.device}}
   Variable: {{$v.Labels.instance}}
   Alerting value: {{ $v }}
{{ end }}
{{ end }}

^^ device and instance may vary, based on query returned values(labels)

And the result i get:

    Alert summary:
     
       1 firing:
       
     [firing] EKS Storage
     
      Location: /dev/nvme0n1p1
      Variable: 10.0.112.170:9100
      Alerting value: 24.88013967279807
    ```

Chitterlings