I'm currently building some Grafana managed alerts on Grafana v8.0.3 using Loki as my datasource.
I have the following query (A):
count_over_time({type="ErrorLogs",fullPipelineName="Test"} [20s])
And I have an expression (B) which triggers the alert when the sum of A is above 0 for 10s.
The alerts seem to fire as expected, which is basically as any log comes into Loki with the above labels. However, these logs also have a variety of other labels attached which I want to use in the alerts labels and annotations, for example they have a reason and a resolution label. But I cannot figure out how to access these values, or if it's even possible. I've tried every permutation of things like {{ $labels.reason }}, {{ reason }}, {{ $reason }}, etc. but nothing seems to work.
In case I'm going about this completely wrong and someone could provide a better query or way of doing this, ultimately what I want to do is:
Create an alert whenever a log comes into Loki that matches some labels and send it to OpsGenie. The alert should be created immediately (or ASAP) after the log comes in and an alert should be created for each matching log (which is a potential problem with my above query as if two matching logs come through within a 20s window the alert will only fire once, although this is unexpected behaviour in this case). I want to pass through other label values from the log (labels not used in the query) with the alert to OpsGenie.
Any help would be appreciated.