how to create alert per error in stackdriver

Question

Having created log-based metrics in cloud console, I then want to create alerts so that every time there is a new matching log entry, the alert triggers.

In trying to create a suitable metric, the most likely looking options seem to be threshold or rate of change, but I don't think either will work for a policy of 1 log message => 1 alert.

Help appreciated.

Galimatias

hi, have you solved this somehow? I am trying some hacks with the Conditions but I am not even close to this specific behavior. Thanks! – Geoponics

I have shared our experience in this answer, which might be helpful for others interested in this question. – Sporophore

Answer 1

Yes, today the only way to alert on the log message is to create a threshold condition on the log metric with a very small threshold (0.001) and a duration of 1 minute. Thanks for using Stackdriver.

Nix

This is really disappointing. Perhaps there should be a warning that logging and log fitlers aren't married up to alert polices yet. I struggled with all sorts of random combo to get alerts when there are more than 100 specific messages in a minute with nothing working reliably. – Gunmaker

I just want to trigger alert on every incoming log entry with severity X, why this is so complicated? (metrics, thresholds, periods, etc) – Loving

Answer 2

You can use another alert triggering software (like PagerDuty) which is pinged by emails sent by Stackdriver. PagerDuty is able to filter all those emails that has the word RESOLVE in its subject. They can be absolutely thrown away in our case if you'd like to avoid autoresolving. Of course Stackdriver and PagerDuty alerts will diverge from each other (states will be inconsistent) but you should consider PD as single source of truth this time. It could be a possible workaround.

Philbert

Answer 3

With log-based alerts, we can create alerts from the logs, incident will be creatied for each matching entry.

https://cloud.google.com/blog/products/operations/create-logs-alerts-preview

Hankhanke