How can I find the source of my Hot LRS Write Operations on Azure Storage Account?

Question

We are using an Azure Storage account to store some files that shall be downloaded by our app on the users demand.

Even though there should be no write operations (at least none I could think of), we are exceeding the included write operations just some days into the billing period (see image).

Regarding the price it's still within limits, but I'd still like to know whether this is normal and how I can analyze the matter. Besides the storage we are using

• Functions and
• App Service (mobile app)

but none of them should cause that many write operations. I've checked the logs of our functions and none of those that access the queues or the blobs have been active lately. There are are some functions that run every now and then, but only once every few minutes and those do not access the storage at all.

I don't know if this is related, but there is a kind of periodic ingress on our blob storage (see the image below). The period is roundabout 1 h, but there is a baseline of 100 kB per 5 min.

Analyzing the metrics of the storage account further, I found that there is a constant stream of 1.90k transactions per hour for blobs and 1.3k transactions per hour for queues, which seems quite exceptional to me. (Please not that the resolution of this graph is 1 h, while the former has a resolution of 5 minutes)

Is there anything else I can do to analyze where the write operations come from? It kind of bothers me, since it does not seem as if it's supposed to be like that.

Feverish

Have you had any luck on finding the source of this problem? I'm about to believe that it's not directly to storage accounts, but to automatic backups, will have some testing tomorrow. – Gearalt

Answer 1

I 've had the exact same problem; after enabling Storage Analytics and inspecting the $logs container I found many log entries that indicate that upon every request towards my Azure Functions, these write operations occur against the following container object:

https://[function-name].blob.core.windows.net:443/azure-webjobs-hosts/locks/linkfunctions/host?comp=lease

In my Azure Functions code I do not explicitly write in any of container or file as such but I have the following two Application Settings configured:

• AzureWebJobsDashboard
• AzureWebJobsStorage

So I filled a support ticker in Azure with the following questions:

1. Are the write operation triggered by these application settings? I believe so but could you please confirm.
2. Will the write operation stop if I delete these application settings?
3. Could you please describe, in high level, in what context these operations occur (e.g. logging? resource locking, other?)

and I got the following answers from Azure support team, respectively:

1. Yes, you are right. According to the logs information, we can see “https://[function-name].blob.core.windows.net:443/azure-webjobs-hosts/locks/linkfunctions/host?comp=lease”. This azure-webjobs-hosts folder is associated with function app and it’s created by default as well as creating function app. When function app is running, it will record these logs in the storage account which is configured with AzureWebJobsStorage.
2. You can’t stop the write operations because these operations record necessary logs to storage account used by Azure Functions runtime. Please do not remove application setting AzureWebJobsStorage. The Azure Functions runtime uses this storage account connection string for all functions except for HTTP triggered functions. Removing this Application Settings will cause your function app unable to start. By the way, you can remove AzureWebJobsDashboard and it will stop Monitor rather than the operation above.
3. These operations is to record runtime logs of function app. These operations will occur when our backend allocates instance for running the function app.

Roguery

In summary, this is one of the costs for running Azure Functions. (that isn't obvious) No different than Function Runtime Cost. – Sinistrad

Answer 2

Best place to find information about storage usage is to make use of Storage Analytics especially Storage Analytics Logging.

There's a special blob container called $logs in the same storage account which will have detailed information about every operation performed against that storage account. You can view the blobs in that blob container and find the information.

If you don't see this blob container in your storage account, then you will need to enable storage analytics on your storage account. However considering you can see the metrics data, my guess is that it is already enabled.

Regarding the source of these write operations, have you enabled diagnostics for your Functions and App Service? These write diagnostics logs to blob storage. Also, storage analytics is also writing to the same account and that will also cause these write operations.

Accessary

I'm facing this exact same issue, never had it before, nothing new on Storage Analytics Logging, every operation logged is stored as Cold and its aligned with my app log, can't find the source of this LRS write operation, my only guess its that some hidden LRS operation that is not public acessible from azure portal like database backups are being done with more frequency, can't know for sure, any other tips? – Gearalt

@PauloLima I too am facing this issue and opened another post. Have you had any luck fixing this on your end? I've had no luck so far stackoverflow.com/questions/57654009 – Gizmo

@MarkB nothing here, i've an open ticket on Azure Support Team for quite some time (a month maybe) and they can't provide me with an answer to this question, neither they were able to filter on their end for hot blobs. – Gearalt

@MarkB turns out that today i've received an answer: "Files that are accessed constantly will show up as Hot even if they are setup as cool." – Gearalt

@PauloLima I too have a ticket open with MSFT. nice to see that you finally got a reply, albeit a bit useless I think! The funny thing on my end is, that i'm not using file storage at all! Which begs to ask the question... WHAT IS BEING WRITTEN TO MY FILES! – Gizmo

@MarkB that's serious, if I've any other updates i'll keep you informed. – Gearalt

Answer 3

For my case, I have a Azure App Insight which took 10K transactions on its storage per mintues for functions and app services, even thought there are only few https requests among them. I'm not sure what triggers them, but once I removed app insights, everything becomes normal.

Ashliashlie