Azure Storage Blob Rename

Question

Is is possible to rename an Azure Storage Blob using the Azure Storage API from a Web Role? The only solution I have at the moment is to copy the blob to a new blob with the correct name and delete the old one.

Depraved

Now, yes using ADLS Gen 2 with the hierarchical namespace - learn.microsoft.com/en-us/azure/storage/data-lake-storage/… – Venusian

Answer 1

There is practical way to do so, although Azure Blob Service API does not directly support ability to rename or move blobs.

Alkmaar

Try ADLS Gen 2 API where you can Rename blob just like you rename a file in an atomic operation. - azure.microsoft.com/en-us/services/storage/data-lake-storage – Venusian

@SaherAhwal file resource is not a blob resource. There is no API so far to rename a blob in an atomic operation – Uribe

@Uribe yes that's why you need to use ADLS Gen 2 and enable Hierarchical Namespace. – Venusian

You can now do this, see stackoverflow.com/revisions/38973244/4 – Sweyn

@ErikErikson that is renaming a container, not a blob. – Pompei

What's more, a later revision of that comment reports the feature rescinded. – Sweyn

Answer 2

UPDATE:

I updated the code after @IsaacAbrahams comments and @Viggity's answer, this version should prevent you from having to load everything into a MemoryStream, and waits until the copy is completed before deleting the source blob.

---

For anyone getting late to the party but stumbling on this post using Azure Storage API V2, here's an extension method to do it quick and dirty (+ async version):

public static class BlobContainerExtensions 
{
   public static void Rename(this CloudBlobContainer container, string oldName, string newName)
   {
      //Warning: this Wait() is bad practice and can cause deadlock issues when used from ASP.NET applications
      RenameAsync(container, oldName, newName).Wait();
   }

   public static async Task RenameAsync(this CloudBlobContainer container, string oldName, string newName)
   {
      var source = await container.GetBlobReferenceFromServerAsync(oldName);
      var target = container.GetBlockBlobReference(newName);

      await target.StartCopyFromBlobAsync(source.Uri);

      while (target.CopyState.Status == CopyStatus.Pending)
            await Task.Delay(100);

      if (target.CopyState.Status != CopyStatus.Success)
          throw new Exception("Rename failed: " + target.CopyState.Status);

      await source.DeleteAsync();
    }
}

Update for Azure Storage 7.0

    public static async Task RenameAsync(this CloudBlobContainer container, string oldName, string newName)
    {
        CloudBlockBlob source =(CloudBlockBlob)await container.GetBlobReferenceFromServerAsync(oldName);
        CloudBlockBlob target = container.GetBlockBlobReference(newName);


        await target.StartCopyAsync(source);

        while (target.CopyState.Status == CopyStatus.Pending)
            await Task.Delay(100);

        if (target.CopyState.Status != CopyStatus.Success)
            throw new Exception("Rename failed: " + target.CopyState.Status);

        await source.DeleteAsync();            
    }

Disclaimer: This is a quick and dirty method to make the rename execute in a synchronous way. It fits my purposes, however as other users noted, copying can take a long time (up to days), so the best way is NOT to perform this in 1 method like this answer but instead:

• Start the copy process
• Poll the status of the copy operation
• Delete the original blob when the copy is completed.

Middendorf

@BrianMacKay mentioned that the StartCopyFromBlob may take 7 days to complete. Is there any truth to that as far as you know? – Whom

Hi @Paqogomez, maybe according to the SLA somewhere, but in my experience it's fast (in the milliseconds to seconds range) – Middendorf

AFAIK StartCopyFromBlob will return when the copy operation has STARTED. It will not return when the copy is complete! To identify when the copy operation is done you need to poll the latest properties of the blob and see when the copy operation is complete. – Dualism

@IsaacAbraham are you suggesting it could be deleted before it's copied? I have not had a problem with this method at all, and the files are moved fast. – Middendorf

@Zidad: That's exactly what I'm suggesting. It /should/ happen quickly, and generally will do. But you can't guarantee this, particularly if you are moving lots of files - Azure seems to throttle the number of concurrent transfers any one account can do at a time after a while, so you could see items queued for transfer. I'm wondering whether the delete operation might fail though, because the Copy operation would have theoretically taken out a lock on the file. – Dualism

In a high load situation, I lost approx 20% of the files I was renaming because the delete beat the copy. The operations DO NOT QUEUE. Fix here: https://mcmap.net/q/293618/-azure-storage-blob-rename/… – Goosefoot

I've seen the copying take more than 24h several times. So please don't use this method if you want to stay sane. The copying is performed in the background with low priority in the Azure network. The copying isn't throttled per account, meaning your copy performance may differ depending on other Azure customers. – Fairground

The throttling algorithm is not publicised but I have definitely seen per-account throttling in the past. But in addition to that, yes, it's also bound to available resources on the Azure network too. – Dualism

CopyState is apparently always null... following is the code i have var blob = (CloudBlockBlob)_root.CloudContainer.GetBlobReferenceFromServer(blobName); var blobCopy = _root.CloudContainer.GetBlockBlobReference(newBlobName); if (blobCopy.ExistsAsync().Result) return Json(response); if (!blob.ExistsAsync().Result) return Json(response); blobCopy.StartCopyAsync(blob); while (blobCopy.CopyState.Status == CopyStatus.Pending) Task.Delay(100); blob.DeleteIfExistsAsync(); – Horticulture

@Horticulture If I look quickly, you're not using 'await' on asynchronous operations. I think you should first look into the basics of async/await constructs in C# and then ask a new question (link here if you want me to answer it), that should give you a better answer. – Middendorf

Answer 3

I originally used code from @Zidad, and in low load circumstances it usually worked (I'm almost always renaming small files, ~10kb).

DO NOT StartCopyFromBlob then Delete!!!!!!!!!!!!!!

In a high load scenario, I LOST ~20% of the files I was renaming (thousands of files). As mentioned in the comments on his answer, StartCopyFromBlob just starts the copy. There is no way for you to wait for the copy to finish.

The only way for you to guarantee the copy finishes is to download it and re-upload. Here is my updated code:

public void Rename(string containerName, string oldFilename, string newFilename)
{
    var oldBlob = GetBlobReference(containerName, oldFilename);
    var newBlob = GetBlobReference(containerName, newFilename);

    using (var stream = new MemoryStream())
    {
        oldBlob.DownloadToStream(stream);
        stream.Seek(0, SeekOrigin.Begin);
        newBlob.UploadFromStream(stream);

        //copy metadata here if you need it too

        oldBlob.Delete();
    }
}

Goosefoot

Hi Viggity, thanks, it seems like @IsaacAbraham was right, sorry about that. I've updated my answer with a warning. – Middendorf

Apparently there is a status you can check as well which would allow you to rename the blob without having to download it all into memory like you do, I'll update my answer... – Middendorf

@zidad, interesting approach. in my particular case, I don't want to make it async as it'd hose a couple of other things. Thanks for the update. – Goosefoot

Why do people abuse MemoryStreams like this? It's horrible. You can pipe the oldBlob stream directly into the newBlob stream, using a small byte buffer. This entirely beats the purpose of Stream and is totally unscalable. I've seen this same code copied into production and it's caused OOM issues. Terrible. – Adachi

all my files were small, didn't matter. sorry – Goosefoot

@Al-Muhandis please post an alternative answer. – Bulbar

Answer 4

You can, however, copy and then delete.

Chuppah

if you copy, make sure you copy the actual data and metadata, THEN delete. DO NOT use StartCopyFromBlob and then Delete. I lost 20% of my files I was renaming because the Copy did not finish before the Delete took effect. https://mcmap.net/q/293618/-azure-storage-blob-rename/… – Goosefoot

Answer 5

While this is an old post, perhaps this excellent blog post will show others how to very quickly rename blobs that have been uploaded.

Here are the highlights:

//set the azure container
string blobContainer = "myContainer";
//azure connection string
string dataCenterSettingKey = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", "xxxx",
                                            "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx");
//setup the container object
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(dataCenterSettingKey);
CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(blobContainer);

// Set permissions on the container.
BlobContainerPermissions permissions = new BlobContainerPermissions();
permissions.PublicAccess = BlobContainerPublicAccessType.Blob;
container.SetPermissions(permissions);

//grab the blob
CloudBlob existBlob = container.GetBlobReference("myBlobName");
CloudBlob newBlob = container.GetBlobReference("myNewBlobName");
//create a new blob
newBlob.CopyFromBlob(existBlob);
//delete the old
existBlob.Delete();

Whom

Consider updating your answer with a brief summary of the relevant points. Then it will be still be useful to others, even if the link breaks. – Colincolinson

As far as I can tell, this is out of date as of Azure Storage API 2... Now CopyFromBlob has become StartCopyFromBlob, and I'm hearing that this puts your blob in a rename queue that can take at most 7 days! – Bioastronautics

that has not been my experience. While it may say something like that in the SLA, its very fast to create and copy. – Whom

The v2 api turns CopyFromBlob into StartCopyFromBlob. In a high load situation I lost 20% of the files I was renaming. NOT GOOD. https://mcmap.net/q/293618/-azure-storage-blob-rename/… – Goosefoot

@paqogomez - in reality performance varies depending on other Azure customers. I've been in contact with Microsoft and they have confirmed that sometimes the copying takes days. – Fairground

Answer 6

Renaming is not possible. Here is a workaround using Azure SDK for .NET v12:

BlobClient sourceBlob = container.GetBlobClient(sourceBlobName);
BlobClient destBlob = container.GetBlobClient(destBlobName);
CopyFromUriOperation ops = await destBlob.StartCopyFromUriAsync(sourceBlob.Uri);

long copiedContentLength = 0;
while (ops.HasCompleted == false)
{
    copiedContentLength = await ops.WaitForCompletionAsync();
    await Task.Delay(100);
}
await sourceBlob.DeleteAsync();

Archy

Answer 7

Copy the blob, then delete it.

Tested for files of 1G size, and it works OK.

For more information, see the sample on MSDN.

StorageCredentials cred = new StorageCredentials("[Your?storage?account?name]", "[Your?storage?account?key]");  
CloudBlobContainer container = new CloudBlobContainer(new Uri("http://[Your?storage?account?name].blob.core.windows.net/[Your container name] /"), cred);  

string fileName = "OldFileName";  
string newFileName = "NewFileName";  
await container.CreateIfNotExistsAsync();  

CloudBlockBlob blobCopy = container.GetBlockBlobReference(newFileName);  

if (!await blobCopy.ExistsAsync())  
{  
    CloudBlockBlob blob = container.GetBlockBlobReference(fileName);  

    if (await blob.ExistsAsync())  
    {  
           // copy
           await blobCopy.StartCopyAsync(blob);                               
           // then delete
           await blob.DeleteIfExistsAsync();  
    } 
} 

Molest

Answer 8

You can now with the new release in public preview of ADLS Gen 2 ( Azure Data Lake Storage Gen 2)

The Hierarchical Namespace capability allows you to perform atomic manipulation of directories and files which includes Rename operation.

However, make note of the following: "With the preview release, if you enable the hierarchical namespace, there is no interoperability of data or operations between Blob and Data Lake Storage Gen2 REST APIs. This functionality will be added during preview."

You will need to make sure you create the blobs (files ) using ADLS Gen 2 to rename them. Otherwise, wait for the interoperability between Blob APIs and ADLS Gen 2 to be added during the preview time period.

Venusian

Answer 9

Using Monza Cloud's Azure Explorer, I can rename an 18 Gigabyte blob in under a second. Microsoft's Azure Storage Explorer takes 29 sec to clone that same blob, so Monza is not doing a copy. I know it is fast because immediately after the Monza rename, clicking the container in Microsoft Azure Storage Explorer shows the blob with the new name.

Becoming

Answer 10

The only way at the mement is to move the src blob to a new destination/name. Here is my code to do this

 public async Task<CloudBlockBlob> RenameAsync(CloudBlockBlob srcBlob, CloudBlobContainer destContainer,string name)
    {
        CloudBlockBlob destBlob;

        if (srcBlob == null && srcBlob.Exists())
        {
            throw new Exception("Source blob cannot be null and should exist.");
        }

        if (!destContainer.Exists())
        {
            throw new Exception("Destination container does not exist.");
        }

        //Copy source blob to destination container            
        destBlob = destContainer.GetBlockBlobReference(name);
        await destBlob.StartCopyAsync(srcBlob);
        //remove source blob after copy is done.
        srcBlob.Delete();
        return destBlob;
    }

Here is a code sample if you want the blob lookup as part of the method:

    public CloudBlockBlob RenameBlob(string oldName, string newName, CloudBlobContainer container)
    {
        if (!container.Exists())
        {
            throw new Exception("Destination container does not exist.");
        }
        //Get blob reference
        CloudBlockBlob sourceBlob = container.GetBlockBlobReference(oldName);

        if (sourceBlob == null && sourceBlob.Exists())
        {
            throw new Exception("Source blob cannot be null and should exist.");
        }

        // Get blob reference to which the new blob must be copied
        CloudBlockBlob destBlob = container.GetBlockBlobReference(newName);
        destBlob.StartCopyAsync(sourceBlob);

        //Delete source blob
        sourceBlob.Delete();
        return destBlob;
    }

Auditor

Answer 11

There is also a way without copying your blob to rename it, and without running any script: mounting Azure Blob storage to your OS: https://learn.microsoft.com/bs-latn-ba/azure/storage/blobs/storage-how-to-mount-container-linux

Then you can just use mv and your blob will be renamed instantly.

Cypripedium

Answer 12

Using Azure Storage Explorer is the easiest way to manually rename a blob. You can download it here https://azure.microsoft.com/en-us/features/storage-explorer/#overview

Cardew

Answer 13

If you set the ContentDisposition property with

attachment;filename="yourfile.txt"

The name of the download over http will be whatever you want.

I think Storage was built with the assumption that data would be stored in a way with unique identifiers primarily used as the filenames. Issuing Shared Access Signatures for all downloads is a bit weird though, so this isn't ideal for some people.

But I think abstracting away the user-facing filename is overall a good practice and encourages a more stable architecture overall.

Oneida

I downvoted this because this answer doesn't actually rename the blob, which is what the OP asked for. While using Content-Disposition is a clever trick, it isn't a real solution. – Whish

Answer 14

This worked for me in live environment of 100K Users having file sizes no more than 100 mb. This is similar synchronous approach to @viggity's answer. But the difference is that its copying everything on Azure side so you don't have to hold Memorystream on your server for Copy/Upload to new Blob.

 var account = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(StorageAccountName, StorageAccountKey), true);
 CloudBlobClient blobStorage = account.CreateCloudBlobClient();
 CloudBlobContainer container = blobStorage.GetContainerReference("myBlobContainer");

 string fileName = "OldFileName";  
 string newFileName = "NewFileName"; 

 CloudBlockBlob oldBlob = container.GetBlockBlobReference(fileName);
 CloudBlockBlob newBlob = container.GetBlockBlobReference(newFileName);
 using (var stream = new MemoryStream())
 {
      newBlob.StartCopyFromBlob(oldBlob);
      do { } while (!newBlob.Exists());
      oldBlob.Delete();
 }

Capitulary

Why are you using a new MemorySteam that is never referenced? – Amalberga

Sorry, I forked that code from @viggity's answer and forgot to remove that reference. But you can get an idea of how it can be done at server side. – Capitulary