Copying one Azure blob to another blob in Azure Storage Client 2.0

Question

In the old 1.7 storage client there was a CloudBlob.CopyFromBlob(otherBlob) method, but it does not seem to be present in the 2.0 version. What is the recommended best practice for copying blobs? I do see a ICloudBlob.BeginStartCopyFromBlob method. If that is the appropriate method, how do I use it?

Prospect

Answer 1

Gaurav Mantri has written a series of articles on Azure Storage on version 2.0. I have taken this code extract from his blog post of Storage Client Library 2.0 – Migrating Blob Storage Code for Blob Copy

CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(accountName, accountKey), true);
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer sourceContainer = cloudBlobClient.GetContainerReference(containerName);
CloudBlobContainer targetContainer = cloudBlobClient.GetContainerReference(targetContainerName);
string blobName = "<Blob Name e.g. myblob.txt>";
CloudBlockBlob sourceBlob = sourceContainer.GetBlockBlobReference(blobName);
CloudBlockBlob targetBlob = targetContainer.GetBlockBlobReference(blobName);
targetBlob.StartCopyFromBlob(sourceBlob);

Posey

I'm trying to use this to make a copy of a blob within the same storage account but I'm getting a 409 CONFLICT error. The message is that "The blob type is invalid for this operation". I've tried treating it as both a Page blob and a Block blob (I'm pretty sure i's a Page blob) – Wall

Resolved it. Turns out the source was a Page blob, but the destination had already been created (in a previous failed attempt) as a Block blob. Deleting the destination and trying again worked. – Wall

Is this server side copy? – Fimbria

Answer 2

Using Storage 6.3 (much newer library than in original question) and async methods use StartCopyAsync (MSDN)

  CloudStorageAccount storageAccount = CloudStorageAccount.Parse("Your Connection");

  CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
  CloudBlobContainer container = blobClient.GetContainerReference("YourContainer");

  CloudBlockBlob source = container.GetBlockBlobReference("Your Blob");
  CloudBlockBlob target = container.GetBlockBlobReference("Your New Blob");
        
  await target.StartCopyAsync(source);

Gibe

Answer 3

FYI as of the latest version (7.x) of the SDK this no longer works because the BeginStartCopyBlob function no longer exists.

You can do it this way:

// this tunnels the data via your program,
// so it reuploads the blob instead of copying it on service side
using (var stream = await sourceBlob.OpenReadAsync())
{
  await destinationBlob.UploadFromStreamAsync(stream);
}

As mentioned by @(Alexey Shcherbak) this is a better way to proceed:

await targetCloudBlob.StartCopyAsync(sourceCloudBlob.Uri);

while (targetCloudBlob.CopyState.Status == CopyStatus.Pending)
{
    await Task.Delay(500);
    // Need to fetch or "CopyState" will never update
    await targetCloudBlob.FetchAttributesAsync(); 
}

if (targetCloudBlob.CopyState.Status != CopyStatus.Success)
{
    throw new Exception("Copy failed: " + targetCloudBlob.CopyState.Status);
}

Death

this will re-upload the blob. As Aaron Sherman mentioned - with newer library one should use StartCopyAsync – Thurlough

If you could add Vangaorth's FetchAttributes() amendment inline that might be useful - I hadn't seen until after a fair bit of debugging. – Czarevna

Starting Windows Azure Storage 8.0 you need to constantly fetch attributes as well ( targetCloudBlob.FetchAttributes() ). Snippet above will fail due timeout (endless loop). – Epifaniaepifano

targetCloudBlob.StartCopyAsync(sourceCloudBlob.Uri) needs to be awaited. – Desuetude

Answer 4

Starting Azure Storage 8, to move Blobs between Storage Accounts I use code similar to below, hope it will help somebody:

//copy blobs - from
CloudStorageAccount sourceStorageAccount = new CloudStorageAccount(new StorageCredentials(storageFromName, storageFromKey), true);
CloudBlobClient sourceCloudBlobClient = sourceStorageAccount.CreateCloudBlobClient();
CloudBlobContainer sourceContainer = sourceCloudBlobClient.GetContainerReference(containerFromName);

//copy blobs - to
CloudStorageAccount targetStorageAccount = new CloudStorageAccount(new StorageCredentials(storageToName, storageToKey), true);
CloudBlobClient targetCloudBlobClient = targetStorageAccount.CreateCloudBlobClient();
CloudBlobContainer targetContainer = targetCloudBlobClient.GetContainerReference(containerToName);

//create target container if didn't exists
try{
    await targetContainer.CreateIfNotExistsAsync();
}
catch(Exception e){
    log.Error(e.Message);
}

CloudBlockBlob sourceBlob = sourceContainer.GetBlockBlobReference(blobName);
CloudBlockBlob targetBlob = targetContainer.GetBlockBlobReference(blobName);

try{
    //initialize copying
    await targetBlob.StartCopyAsync(sourceBlob.Uri);
}
catch(Exception ex){
    log.Error(ex.Message);
    //return error, in my case HTTP
    return req.CreateResponse(HttpStatusCode.BadRequest, "Error, source BLOB probably has private access only: " +ex.Message);
} 

//fetch current attributes
targetBlob.FetchAttributes();

//waiting for completion
while (targetBlob.CopyState.Status == CopyStatus.Pending){
    log.Info("Status: " + targetBlob.CopyState.Status);
    Thread.Sleep(500);
    targetBlob.FetchAttributes();
}

//check status
if (targetBlob.CopyState.Status != CopyStatus.Success){
    //return error, in my case HTTP
    return req.CreateResponse(HttpStatusCode.BadRequest, "Copy failed with status: " + targetBlob.CopyState.Status);
}

//finally remove source in case Copy Status was Success
sourceBlob.Delete();

//and return success (in my case HTTP)
return req.CreateResponse(HttpStatusCode.OK, "Done.");

Epifaniaepifano

Answer 5

It seems that the API might have been cleaned up a little since previous posts were made.

// _client is a BlobServiceClient injected via DI in the constructor.

BlobContainerClient sourceContainerClient = _client.GetBlobContainerClient(sourceContainerName);
BlobClient sourceClient = sourceContainerClient.GetBlobClient(blobName);

BlobContainerClient destContainerClient = _client.GetBlobContainerClient(destContainerName);
BlobClient destClient = destContainerClient.GetBlobClient(blobName);

// assume that if the following doesn't throw an exception, then it is successful.
CopyFromUriOperation operation = await destClient.StartCopyFromUriAsync(sourceClient.Uri);
await operation.WaitForCompletionAsync();

The documentation for operation.WaitForCompletionAsync says:

Periodically calls the server till the long-running operation completes. This method will periodically call UpdateStatusAsync till HasCompleted is true, then return the final result of the operation.

Reviewing the source code for this method seems to call BlobBaseClient.GetProperties (or the async version) which will throw an RequestFailureException on error.

Caniff

Thanks. This works well for making sure the file was copied in real time. This is wonderful WaitForCompletionAsync(); – Trimmer

Answer 6

Naveen already explained the correct syntax for using StartCopyFromBlob (the synchronous method). The method you mentioned (BeginStartCopyFromBlob) is the asynchronous alternative which you can use in combination with a Task for example:

    var blobClient = account.CreateCloudBlobClient();

    // Upload picture.
    var picturesContainer = blobClient.GetContainerReference("pictures");
    picturesContainer.CreateIfNotExists();
    var myPictureBlob = picturesContainer.GetBlockBlobReference("me.png");
    using (var fs = new FileStream(@"C:\Users\Public\Pictures\Sample Pictures\Chrysanthemum.jpg", FileMode.Open))
        myPictureBlob.UploadFromStream(fs);

    // Backup picture.
    var backupContainer = blobClient.GetContainerReference("backup");
    backupContainer.CreateIfNotExists();
    var backupBlob = picturesContainer.GetBlockBlobReference("me.png");

    var task = Task.Factory.FromAsync<string>(backupBlob.BeginStartCopyFromBlob(myPictureBlob, null, null), backupBlob.EndStartCopyFromBlob);
    task.ContinueWith((t) =>
    {
        if (!t.IsFaulted)
        {
            while (true)
            {
                Console.WriteLine("Copy state for {0}: {1}", backupBlob.Uri, backupBlob.CopyState.Status);
                Thread.Sleep(500);
            }
        }
        else
        {
            Console.WriteLine("Error: " + t.Exception);
        }
    });

Elise

StartCopyFromBlob is also an asynchronous operation because copy blob operation is asynchronous – Vanvanadate

Answer 7

For me, WindowsAzure.Storage 8.0.1, James Hancock's solution did the server side copy but the client copy status was stuck on Pending (looping forever). Solution was to call FetchAttributes() on targetCloudBlob after Thread.sleep(500).

// Aaron Sherman's code 

targetCloudBlob.StartCopy(sourceCloudBlob.Uri);

while (targetCloudBlob.CopyState.Status == CopyStatus.Pending)
{
    Thread.Sleep(500);
    targetCloudBlob.FetchAttributes();
}

// James Hancock's remaining code

Official Microsoft documentation (async example)

Ariel

Answer 8

here is my short simple answer.

public void Copy(CloudBlockBlob srcBlob, CloudBlobContainer destContainer)
{
    CloudBlockBlob destBlob;

    if (srcBlob == null)
    {
        throw new Exception("Source blob cannot be null.");
    }

    if (!destContainer.Exists())
    {
        throw new Exception("Destination container does not exist.");
    }

    //Copy source blob to destination container
    string name = srcBlob.Uri.Segments.Last();
    destBlob = destContainer.GetBlockBlobReference(name);
    destBlob.StartCopyAsync(srcBlob);                
}

Forbis