Prometheus Slack alert with FAQ url

Question

I have a Prometheus alert like this:

- alert: NginxCrashLoop
  annotations:
    description: Nginx at {{ $labels.pod }} is in Crash Loop
    identifier: '{{ $labels.node }}'
    runbook_url: https://sites.google.com/a/domain.com/bi/faq/nginx
  expr: |
    rate(kube_pod_container_status_restarts_total{container="frontend"}[2m]) * on(namespace, pod) group_left(node) kube_pod_info * 3600 > 3
  labels:
    severity: critical

and a Go template for Slack custom alerting:

{{ define "__single_message_title" }}{{ range .Alerts.Firing }}{{ .Labels.alertname }} at {{ .Annotations.identifier }}{{ end }}{{ range .Alerts.Resolved }}{{ .Labels.alertname }} at {{ .Annotations.identifier }}{{ end }}{{ end }}

{{ define "custom_title" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ if or (and (eq (len .Alerts.Firing) 1) (eq (len .Alerts.Resolved) 0)) (and (eq (len .Alerts.Firing) 0) (eq (len .Alerts.Resolved) 1)) }}{{ template "__single_message_title" . }}{{ end }}{{ end }}

{{ define "faqURL" }}{{ range .Alerts.Firing }}{{ .Annotations.runbook_url }}
{{ end }}{{ end }}

{{ define "custom_slack_message" }}
{{ if or (and (eq (len .Alerts.Firing) 1) (eq (len .Alerts.Resolved) 0)) (and (eq (len .Alerts.Firing) 0) (eq (len .Alerts.Resolved) 1)) }}
{{ range .Alerts.Firing }}{{ .Annotations.description }}{{ end }}{{ range .Alerts.Resolved }}{{ .Annotations.description }}{{ end }}
{{ else }}
{{ if gt (len .Alerts.Firing) 0 }}
*Alerts Firing:*
{{ range .Alerts.Firing }}- {{ .Annotations.identifier }}: {{ .Annotations.description }}: <{{ template "faqURL" . }}|Quick FAQ>
{{ end }}{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
*Alerts Resolved:*
{{ range .Alerts.Resolved }}- {{ .Annotations.identifier }}: {{ .Annotations.description }}
{{ end }}{{ end }}
{{ end }}
{{ end }}

I wish it was able to send me a link to a FAQ documentation to mitigate the problem, shown after the description. But, the slack message still doesn't contain the link. What am I doing wrong?

Blasting

Any clue, folks? – Blasting

Can you paste your alertmanager.yml as well? – Whist

Answer 1

Here is how I used

I have a custom template with slack text defined as below

{{`{{ define "slack.custom.text" }}`}}
{{`{{ range .Alerts }}`}}
*Alert:* {{`{{ .Annotations.summary }}`}} - `{{`{{ .Labels.severity }}`}}`
*Description:* {{`{{ .Annotations.description }}`}}
*Runbook:* {{`{{ .Annotations.runbook}}`}}
*Details:*
{{`{{ range .Labels.SortedPairs }}`}} • *{{`{{ .Name }}`}}:* `{{`{{ .Value }}`}}`
{{`{{ end }}`}}
{{`{{ end }}`}}
{{`{{ end }}`}}

and then call this in alert_manager_config like

slack_configs:
      - channel: slack-channel-id
        title: '{{`{{ template "slack.default.title" . }}`}}'
        text: '{{`{{ template "slack.custom.text" . }}`}}'

templates:
- '/etc/alertmanager/config/custom.tmpl'

Warfore