Azure alert only fired once

Question

I have implemented an Azure alert that should fire when a Application Insights metric is greater than zero. The metric is the number of items in a Azure poison queue. The metric is calculated by a Azure Function described in this article: https://www.scaling-to-the-sky.com/2018/03/08/poison-queue-monitoring-with-azure-functions/?unapproved=160&moderation-hash=072116753136d2008f5e63a856d8e4b0#comment-160.

The alert has only fired once despite the condition being met on several occasions. I don't know why it doesn't fire. I have noticed that on the one alert that was fired the monitor condition has never changed from "Fired" to "Resolved". Maybe that is the reason why no need new alerts are fired? If that is the case then how do I change the state of the alert monitor condition?

Turgite

Answer 1

A metric alert that's in a "Fired" state would not trigger again until it's resolved. This is done to reduce noise. Resolution happens automatically after 3 healthy evaluations of your condition (evaluations where the condition isn't breached), and there's no way to manually change the monitor condition to "Resolved".

Can you please confirm if you are sending a metric value on every evaluation of the poison queue, even if the value is 0?

Punch

Hi Harel. In the original question I have attached a screen shot of the log show the custom event and of the condition of the alert. I see now that the "value" of the custom events is 0 but the valueCount is 1. And it looks like the condition of the alert is looking at the aggregation type "Count". – Turgite

I managed to resolve the alert by changing the "Aggregation type" from "Count" to "Total". – Turgite

I have just tried to put a test item in the poison queue and the alert is triggered. So, thanks for your help. – Turgite

Answer 2

Metric alerts are stateful by default, so other alerts aren't fired if there's already a fired alert on a specific time series.

To make a specific metric alert rule stateless and get alerted on every evaluation1 in which the alert condition is met, use one of these options:

• If you create the alert rule programmatically, for example, via Azure Resource Manager, PowerShell, REST, or the Azure CLI, set the autoMitigate property to False.
• If you create the alert rule via the Azure portal, clear the Automatically resolve alerts option under the Alert rule details section.

Check out the details here: Make metric alerts occur every time my condition is met

Ulrick