How Do I Backup Azure Tables and Blobs

Question

I store data in Azure Storage tables and blob storage. I would like to automatically backup my data to protect against accidental data corruption by users or by a software issue. There isn't a good solution for this from Microsoft and, while there are paid solutions for automatic backups, it seems like there should be a straight-forward way to backup and restore this data.

Duplessis

For table storage, make the usage of AzCopy. It works fine and also can be used in automation scripts. azcopy.exe /Source:http://PathToYourTableStorage /Dest:C:\YourFolderPath /Key:StorageAccessKey1or2 – Moorman

You also can use this script in your CI process. – Moorman

I made this node script. It only backup changes to tables so you can run it often. – Blanketing

is there news about this feature one year later? – Bloom

@Serge, check out the answer that I put below to this question. I have updated it with information that I got back from Microsoft support in January 2018. The short version is that, no, there is no better support for table and blob storage backup than there was this time last year. – Duplessis

above scripts a bit outdated and don't work on latest azcopy – Hollah

Answer 1

After doing a lot of research on the best way to automatically backup data, I found that the easiest way to do this is to schedule a job in the Windows Task Scheduler on an Azure virtual machine that uses AZCopy. AZCopy does the work of moving the table data to the VM and then back out to a secondary blob store in Azure to hold the backup. The batch file also causes AZCopy to copy the blobs from the source storage account into the target account directly.

You can see a detailed description of how I made this work complete with links to the batch files that I use to automate the backup at this link: http://www.eastfive.com/2016/03/01/automated-backup-of-azure-storage-tables-and-blobs/

02/08/2018 update on this answer: I raised this question to Azure support on January 18, 2018, assuming that surely they had a way to do this now via a service in Azure. Unfortunately that is not the case and there is (according to Microsoft support) no better way to do table backups than those that existed when this question was first posed. This is the answer from support from 01/24/2018:

"I have reviewed over your case and unfortunately there is not a way to do a snapshot of an entire account or container and so the only way to do snapshots with Azure Storage is through blob iteration and applying snapshots then migrating the snaps over to a secondary account. Or you can simply copy the actual files to another account but this would have higher latency and be more costly on storage capacity, where snapshots would take up less and be quicker to transfer.
The methods we support for transfer are AzCopy and the Data Movement Library which can be used to make custom migration solutions in Java or C#.
If you wanted to automate these processes then you could do this via Powershell and Azure Automation or with Azure Functions but assistance with those solutions would likely need to be raised through another support request as my team purely supports Azure Storage."

So, there still exists no automated way to do these backups. My team is working on a library to do backups. When we have that completed, I will post back here.

05/08/2018 update on this answer: As mentioned, my team has been working on a library to automate backups. You can find the project here: https://github.com/eastfivellc/EastFive.Azure.Storage.Backup. Please feel free to contribute.

10/18/2018 update on this answer: I was able to copy data in Azure using their Azure Data Factory functionality. I used Data Factory to pipe data from my source to target storage for both tables and blobs. However, the data movement costs are exorbitantly high (in the hundreds of dollars per backup). So, this is not a solution for backups. As mentioned in my post from above (and in an answer below), the Azure Data Movement Library is the best solution here.

Duplessis

You might want to re-examine your use of AzCopy, as there is no need to ever copy blobs to an intermediate VM; it's designed to copy directly blob-to-blob. With table storage, you do need to copy data, even if storing json/csv in a blob later. – Wilde

@DavidMakogon, thanks for catching that error in my writing. I have updated the answer to reflect that table data is pulled locally, while blob data is pushed directly to the target. – Duplessis

Sorry -was just clarifying. A long long time ago, blob-copying only worked by transferring to a temporary space first. Didn't know if that (out of data) info was still floating around... – Wilde

Check out this link, David. It has the latest info on AZCopy and clarifies that when copying blobs from a source to target storage account that the copy is done server-side. azure.microsoft.com/en-us/documentation/articles/… – Duplessis

Answer 2

We wrote a .NET library that backups tables and blobs. You can easily implement this in an azure function timer trigger.

In this blog I explain how to implement it using an Azure function.

[FunctionName("Function1")]
public static async Task Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, ILogger log, ExecutionContext context)
{
  var sourceAccountName = Environment.GetEnvironmentVariable("BackupSourceAccountName");
  var sourceKey = Environment.GetEnvironmentVariable("BackupSourceAccountKey");

  var backupAzureStorage = new Luminis.AzureStorageBackup.BackupAzureStorage(sourceAccountName, sourceKey, log, context.FunctionAppDirectory);

  var destinationAccountName = Environment.GetEnvironmentVariable("BackupDestinationAccountName");
  var destinationKey = Environment.GetEnvironmentVariable("BackupDestinationAccountKey");
  var destinationContainerName = Environment.GetEnvironmentVariable("BackupDestinationContainer");

  // Backup Tables
  await backupAzureStorage.BackupAzureTablesToBlobStorage("table1,table2", destinationAccountName, destinationKey, destinationContainerName, "tables");

  // Backup Blobs
  await backupAzureStorage.BackupBlobStorage("container1,container2", destinationAccountName, destinationKey, destinationContainerName, "blobs");
}

Burner

Answer 3

If you are still looking for a solution for this problem, please look at a package I released on NuGet AzureTableUtilities

You can copy a table to a file or blob file, restore a table from a file or blob file as well as copy to another table on the same account or a different account. Filtering on PartitionKey and Timestamp is also available.

I also created a reference command line code base and put it on GitHub which allows this to be executed from a command line or Docker container.

Let me know if this doesn't quite match what you want, and I'll see if I can enhance the functionality. I realize the tool itself does not "automatically" back Azure Tables data up; however, using the command line example either under windows directly or in a Docker container, I think you'd be able to set up whatever process(es) you wanted to back things up.

Burstone

Answer 4

You can also use 3rd party tools like Cerebrata Azure Management Cmdlets or the functionality Asynchronous Copy Blob announced by Microsoft Azure Storage team which will essentially allow you to copy data from one storage account to another storage account without downloading the data locally.

Check the thread for more: What is the best way to backup Azure Blob Storage contents.

Hope this helps.

Ligniform

Derek, Thanks for the comment. I looked at Cerebrata Cmdlets and they looked promising, but they were $125, where my batching out commands to AZCopy.exe is free. Further, if you look at my responses above, you can see that I have clarified that AZCopy does a server-side copy of blobs, so they need never leave Azure. The table storage is a different story - table data must be pulled locally, at least at this point. Thanks for your feedback. – Duplessis

@Keith Holloway, Sorry I didn't notice that and when the thread I attached posted, AzCopy should have not been announced. As of now, AzCopy should be the best way to backup data in Azure. – Ligniform

Cerebrata Azure Management Cmdlets has been discontinued! – Moorman

Answer 5

Have a look at the Microsoft Azure Storage Data Movement Library

This seems to offer most of the functionality of AzCopy, but in a nuget package, so you can use it pretty much anywhere you like.

We are running this in a triggered webjob and it seems to be performing really well for both table and blob storage backup.

Underpinnings

Check out this link from my original answer: eastfive.com/2016/03/01/… We use the same library in a WebJob. I agree with you that, for now, this is the best way to back up AST. One other side note, I tested backups via Azure Data Factory. This will work, but the price for data movement is so high that this is not a viable solution. I concur - DML in a WebJob is the best option. – Duplessis

This is interesting. I almost wanted to implement my own solution for it. – Hornstein

@KeithHolloway I had also wondered about using the Data Factory, so it's good to know that this would work, but not viable :) – Underpinnings

Carl, you say this works for table storage backup, but I don't see anything related to tables on this github. There is an open feature request for table support. How do you use this package for backing up azure tables? – Vernon

Answer 6

You can use Microsoft Azure Storage Explorer to Copy/Paste Tables, Upload/Download file systems, Blob, Cosmos, Data lake etc., its very user friendly tool.

Download Link : https://azure.microsoft.com/en-in/features/storage-explorer/

Hanuman

Azure Storage Explorer from this site is also a useful tool: archive.codeplex.com/?p=azurestorageexplorer . However, the tools do not provide an automated backup, which is what this post covers. – Duplessis