Azure alert on Azure Functions "Failed" metric is triggering with no apparent failures

Question

I want an Azure Alert to trigger when a certain function app fails. I set it up as a GTE 1 threshold on the [function name] Failed metric thinking that would yield the expected result. However, when it runs daily I am getting notifications that the alert fired but I cannot find anything in the Application Insights to indicate the failure and it appears to be running successfully and completing.

Here is the triggered alert summary:

Here is the invocation monitoring from the portal showing that same function over the past few days with no failures:

And here is an application insights search over that time period showing no exceptions and all successful dependency actions:

The question is - what could be causing a Azure Function Failed metric to be registering non-zero values without any telemetry in Application Insights?

Update - here is the alert configuration

And the specific condition settings-

Failures blade for wider time range:

There are some dependency failures on a blob 404 but I think that is from a different function that explicitly checks for the existence of blobs at paths to know which files to download from an external source. Also the timestamps don't fall in the sample period.

No exceptions:

Dotty

Can you share your alert configuration details/screenshot please. – Ardennes

@KrishnenduGhosh-MSFT Sure! I have updated the description with a screenshot of the alert configuration. Thanks – Dotty

Thanks. Would be great if you click that condition and share the screenshot. – Ardennes

@KrishnenduGhosh-MSFT updated again - thanks. – Dotty

Wondering where is the filter condition on the failures only from your 'particular' function in this alert config. I suspect this alert is getting triggered on any failure logged into this application insights instance (I mean includes any other apps/functions also). You can go to your App Insights->"Failure" blade to find out if it's indeed the case . Yes, I saw you shared one screenshot in your concerned time period, but want to be sure it's not an oops of UTC v/s your local time by seeing raw failures :). – Ardennes

@KrishnenduGhosh-MSFT the particulr function in the alert is in the blurred out text there before 'Failures'. When creating the alert I saw one for each function in the function app so I created alerts for each one separately. I don't think it is timezone as the function gets triggered daily at 6pm local (pacific) which is when alert shows up and log shows it ran. Here is the failures blade: – Dotty

@KrishnenduGhosh-MSFT I put the screenshots of the failure blade tabs in the question body – Dotty

Thanks. I am checking. – Ardennes

Ok. I still think the alert is getting fired irrespective of the source of failure (those blob ones). And I am still not clear how you are separating alert for each function (like any custom where condition) learn.microsoft.com/azure/azure-monitor/platform/alerts-log – Ardennes

@KrishnenduGhosh-MSFT it isn't a log alert, it is a metric alert. When I went to create it there are a handful of built in metrics available for each function in my function app. I selected the one the was "[function name] Failures" to use as the metric. pasteboard.co/Jr6kghT.png – Dotty

@KrishnenduGhosh-MSFT and I double checked and don't see how any of those dependency failures were during the time period when this alert triggered. I created this side by side showing the alert trigger information next to a log search for dependency failures and double checked the timezones and you can see that the dependency failures are all around 5AM local time (when a different function does it's processing) while the alert triggered around 6PM local: pasteboard.co/Jr6rhNI.png – Dotty

Thanks. Let me take a deeper look. – Ardennes

I tried to reproduce your scenario, but unfortunately works fine as expected for me. I suggest you to post this in Microsoft Q&A to get a closer troubleshooting from product SMEs. – Ardennes

@GavinH, not sure what's happening in your side as it's ok at my side. But as a workaround, you can use the custom log search to define the alert which is more trustable. – Ventriloquism

Answer 1

Per comment on the question by @ivan-yang I have switched the alerting to use a custom log search instead of the built-in Azure Function metric. At this point that metric seems to be pretty opaque as to what is triggering it and it was triggering every day when I ran the Azure Function with no apparent underlying failure. I plan to avoid this metric now.

My log based alert is using the following query for now to get what I was looking for (an exception happened or a function failed):

requests
| where success == false
| union (exceptions)
| order by timestamp desc

Thanks to @ivan-yang and @krishnendughosh-msft for the help

Dotty

How do you turn this query into an alert rule? I am trying to use it for my Azure Function but there is no custom choice available under "Select a signal". custom log search is not available in the list of signals. – Canakin

I got it, I had to use Application Insights as target resource instead of the function app to access the custom log search – Canakin