How can I monitor number of instances of a function app when it scales out?

Question

I am looking into "Metrics" tab (Platform Features -> Metrics) in Azure portal for my function app. I can see interesting metrics like CPU time, request count, etc. but there is no metric that would show the number of instances that the app has scaled out to.

Is there a way to get the number of instances of the app across time?

Wuhu

Answer 1

One way is to use an App Insights query. This will give you the number of distinct instances running every 30 seconds for the last 24 hours.

You can edit the granularity and the time span as you choose but bear in mind that the larger granularity the less accurate the query will be as instances can spin up and wind down at any time.

let grainTime = 30sec;

traces
| where timestamp >= ago(24h)
| summarize ['rate/minute'] = dcount(cloud_RoleInstance) by bin(timestamp, grainTime)
| render timechart

You can then pin this to your dashboard!

Sheers

Flaws: a) bin() does not autofill to 0. So this does not show when instance count reduces to zero. We can use make_series instead b) More importanly, on push based triggers like eventhub trigger, the function host does not do any activity to be logged into app insights. So app insights does not know whether host is alive or not. – Darrel

I get "no tabular expression statement found" but OssJames' answer below works. – Bisset

@Bisset my guess is that you didn't select the whole query when you ran it. – Sheers

Answer 2

After selecting any metric from the given options we can add another filter. As shown below.

Then we can add the "Instance" property and choose all the instances currently running for the function app. As shown below.

Occlusive

Thanks! That's nice. I also tried, "apply splitting" option next to "add filter", and that is a good view as well. Both give you some idea about the instances, but none allow us to just tack number of instances over time. – Wuhu

When you get this data using Azure API, and apply $filter=Instances eq * you can process the results to get number of instances over time. Shame there is not such a view in Azure Portal. – Rowdyish

Answer 3

As a preview feature, now we can have scale controller emit logs with reasoning to help understand why and how the application have scaled at various points. You will have to add a function configuration as SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose. Then you can query your scale controller logs to know reason and instance count as in this microsoft docs.

I modified the kusto query in the linked document to have a function scaling graph for past 24 hours

traces 
| where customDimensions.Category == "ScaleControllerLogs"
| where customDimensions.Action == "ScaleResult"
| where customDimensions.AppName == "my-function-app-name"
| extend currentInstanceCount = toint(customDimensions.CurrentInstanceCount)
| make-series rawInstanceCounts = max(currentInstanceCount) default=-1 on timestamp in range(ago(24h), now(), 5m)
| extend instanceCountsForwardFilled = series_fill_forward(rawInstanceCounts, -1)
| project timestamp, instanceCountsForwardFilled
| render timechart 

_{You may also emit scale controller logs to Blob Storage. In the above example, I chose AppInsights for quick queries. Also to avoid app insights pricing impact, consider disabling the config parameter once you understood the scaling behaviour.}

Darrel

Will this still log if APPLICATIONINSIGHTS_CONNECTION_STRING app setting is used instead? Docs reference the old instrumentation key setting. I enabled SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose on my app and no logs were products for scale controller and the function definitely scaled according to metrics. – Exteroceptor

Answer 4

traces
| make-series dcount(cloud_RoleInstance) on timestamp from ago(48hour) to now() step 30sec
| render timechart

I combined Sam Barber's answer along with kiranpradeep's reply to get this query, which shows 0 instances correctly.

Calcutta

Hmm but if on elastic premium, we always have a min number of instances and a min number of prewarmed instances. Not sure how to combine that info with this to get a plot of actual running instances at any given time. – Bisset

Answer 5

Steps to do it:

1. Go to the function app in portal
2. Go to the Diagnose and solve problems tab on the lefthand blade
3. In Search for common problems or tools, enter "HTTP Functions Scaling"
4. Scroll down to Number of workers allocated to the Function App

Credit goes to: https://github.com/Azure/Azure-Functions/issues/2323

Biographer

Answer 6

I just found it very easy using the portal. You can switch the view for local and UTC time as well. It tells you that how many instances were running at a particular time for your functiton app. Try this out.

Dorcasdorcea

Only works for the dedicated and maybe consumption plan AFAIK. For elastic, this is not there. – Bisset

Answer 7

Very nice queries by both Sam Barber and OssJames, but I'm not sure they realize their queries will return the total number of instances for all Azure Functions attached to their Application Insights instance. For people that, like me, have a single App Insights instance monitoring many Azure Functions, that won't work.

I improved OssJames query to filter per:

• Azure Function Resource
• Azure Function Method

Here it is:

traces
| where cloud_RoleName == "AzureFunctionResourceName"
| where operation_Name == "AzureFunctionExposedMethodName"
| make-series dcount(cloud_RoleInstance) on timestamp from ago(48hour) to now() step 30sec
| render timechart

Quartziferous