Is there a way to do symbolic links to the blob data when using Azure Storage to avoid duplicate blobs?

Question

I have a situation where a user is attaching files within an application, these files are then persisted to Azure Blob storage, there is a reasonable likelihood that there are going to be duplicates and I want to put in place a solution where duplicate blobs are avoided.

My first thought is to just name the blob as filename_hash but that only captures a subset of duplicates, then filesize_hash was then next thought.

In doing this though it seems like I am losing some of the flexibility of the blob storage to represent the position in a hierarchy of the file, see: Windows Azure: How to create sub directory in a blob container

So I was looking to see if there was a way to create a blob that referenced the blob data i.e. some for of symbolic link but couldn't find what I wanted.

Am I missing something or should I just go with filesize_hash method and store my hierarchy using an alternative method.

Kazachok

Blobs have metadata attribute support, if you need to store some indicator about the origin, but I don't understand what the real problem is. You might be able to get a better response if you elaborate on why the problem arises. – Wilek

Answer 1

No, there's no symbolic links (source: http://social.msdn.microsoft.com/Forums/vi-VN/windowsazuredata/thread/6e5fa93a-0d09-44a8-82cf-a3403a695922).

A good solution depends on the anticipated size of the files and the number of duplicates. If there aren't going to be many duplicates, or the files are small, then it may actually be quicker and cheaper to live with it - $0.15 per gigabyte per month is not a great deal to pay, compared to the development cost! (That's the approach we're taking.)

If it was worthwhile to remove duplicates I'd use table storage to create some kind of redirection between the file name and the actual location of the data. I'd then do a client-side redirect to redirect the client's browser to download the proper version.

If you do this you'll want to preserve the file name (as that will be what's visible to the user) but you can call the "folder" location what you want.

Capri

Thanks, cost of development isn't worth it in this case either. – Kazachok

Answer 2

Another solution to keep all structure of your files but still provide a way to do "symbolic links" could be as follows, but as in the other answer the price might be so small that its not worth the effort of implementing it.

I decided in similar setup to just store the md5 of each uploaded file in a table and then in a year go back and see how many duplicates that got uploaded and how much storage that could be saved. It will at that time make it easy to evaluate if its worth implementing a solution for symbolic links.

The downside of maintaining it all in table storage is that you get a limited query API to your blobs. Instead i would suggest to use the Metadata on blobs for creating links. (meta data turns in to normal headers on the requests when using REST API etc).

So for duplicate blobs, just keep one of them and store a link header telling where the data is.

blob.Metadata.Add("link", dataBlob.Name);
await blob.SetMetadataAsync();
await blob.UploadTextAsync("");

at this point the blob now takes up no data but is still present in storage and will be returned when listing blobs.

Then when accessing data you simply would have to check if a blob has a "link" metadata set or with rest, check if a x-ms-meta-link header is present and then read the data from there instead.

blob.Container.GetBlockBlobReference(blob.Metadata["link"]).DownloadTextAsync()

or any of the other methods for accessing the data.

Above is just the basics and I am sure you can figure out the rest if this is used.

Bewley