What is the best way to monitor a container in Azure blob storage for changes?

Question

I am looking for the best way to monitor a container/folder in Azure blob storage for changes. So far I have only found one way to do this, which is to run a worker process somewhere that pings the container's contents on a regular basis to look for changes.

Is there a better way?

Orphanage

It seems like if you want triggers, you have to use SQL Azure. Here's a nice little comparison: brian.chipsofttech.com/blogs/brian/post/Azure-Data-Storage.aspx – Shawanda

SQL Azure triggers don't apply to blob storage containers, nor does SQL Azure provide the mass-storage capability of blobs and containers. – Cutaneous

Answer 1

The Storage SDK doesn't provide this, but it is a primary feature in the new WebJobs SDK. There's a [BlobInput] attribute that lets you specify a container to listen on, and it includes an efficient blob listener that will dispatch to the method when new blobs are detected. There are some examples of blob listening at: http://blogs.msdn.com/b/jmstall/archive/2014/02/18/azure-storage-bindings-part-1-blobs.aspx

Here's an example usage:

    public static void CopyWithStream(
        [BlobInput("container/in/{name}")] Stream input,
        [BlobOutput("container/out1/{name}")] Stream output
        )
    {
        Debug.Assert(input.CanRead && !input.CanWrite);
        Debug.Assert(!output.CanRead && output.CanWrite);

        input.CopyTo(output);
    }

And the blob listener is under here:

        JobHost host = new JobHost(acs); // From nuget: Microsoft.WindowsAzure.Jobs.Host
        host.RunAndBlock();  

Amylo

It's worth mentioning that the container is being scanned every 10 minutes (!). So the web job won't be triggered immediately. – Empty

Answer 2

As others said, polling in a reasonable interval for your application is OK. But you don't need to check the content itself. You can check ETag (if using plain HTTP) or you can check BlobProperties.LastModifiedUtc if you're using API.

Hama

Ok, that's what I thought. Wouldn't it be nice if there was a push notification type thing for this...? – Orphanage

@BillForney Probably yes, but considering the possibly unlimited size of container and whole distributed nature of it, it's not trivial to do it and to scale well and to not kill infrastructure. – Hama

Is it 100% safe to assume that in case of checking only the BlobProperties.LastModifiedUtc field of a container you will never lose a change? Can't there be any problems for example if multiple machines are used to store a huge container on blob, can't there be an event when there will be a "modification in the past"? – Nephro

It is safe. Once the operation is done/completed successfully, the LastModifiedUtc is updated. – Hama

Answer 3

There is no better way, IMHO. Sorry. HTH

Brothers

Answer 4

Igor's right, in that you'll need to poll the container. I just wanted to clarify something. When you said:

run a worker process somewhere that pings the container's contents on a regular basis to look for changes

The worker process may be a thread in a Web Role or Worker Role, the Run() loop, etc. It doesn't require a separate role. Just make sure your polling code works with multiple role instances (e.g. you may want to do some type of page blob lease as a mutex, to make sure you're polling from only one instance). You can also do this via Azure Queues, since queue messages now support an invisibility timeout upon message-creation.

Cutaneous

Answer 5

Assuming you have control over the code that updates/changes the blob, you can use SignalR to generate a push notification whenever the blob changes thereby avoiding the need for repeated polling.

Weekday

Answer 6

Do the azure storage analytics hold the required information? You would still need something to do the polling.

Turro

Answer 7

Old issue but I was researching this topic and found that you can now use Azure Event Grid (not to be confused with Azure Event Hubs) to get notified when changes in blob storage occurs:

https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview

Ozonolysis