How to Copy All Containers along with Blobs from one Azure storage account to other storage account

Question

I have two Azure Storage accounts. On one of the Storage account, there are almost 100 containers and some blobs in those containers. I want to transfer all these containers along with the blobs in it to other Storage account.

I have seen many tools copying the blobs from one container to other, but didn't come across any that copies the whole containers aswell. Please help with this.

Thanks

Eraser

Answer 1

It's a bit late but this PowerShell script will do it...

#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey

$Containers = Get-AzureStorageContainer -Context $SourceStorageContext

foreach($Container in $Containers)
{
    $ContainerName = $Container.Name
    if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
    {   
        Write-Output "Creating new container $ContainerName"
        New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
    }

    $Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
    $BlobCpyAry = @() #Create array of objects

    #Do the copy of everything
    foreach ($Blob in $Blobs)
    {
       $BlobName = $Blob.Name
       Write-Output "Copying $BlobName from $ContainerName"
       $BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
       $BlobCpyAry += $BlobCopy
    }

    #Check Status
    foreach ($BlobCopy in $BlobCpyAry)
    {
       #Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState but I prefer output with % copied
       $CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
       $Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100) 
       Write-Output $Message
    }
}

Some credit for the script goes to this blog post...

http://windowsitpro.com/azure/copy-content-one-azure-storage-account-another

...I've ammended his code removing nasty long dashes and added code to loop through the containers instead of specifying them manually.

Wallop

Is it intended for the script to end before all copies are done? Also note that the divition (inside the "check status" loop) may cause a "div by zero" error. Also, the first link in the answer is broken. – Lacroix

@Lacroix true! but only if there's no bytes to copy. You can just delete those two lines... they're only informational. Or you could put an if statement – Wallop

The source link is broken btw. Do you know of another mirror of that article? – Manwell

Answer 2

You can try the following script based on Azcopy. It will list all containers in a storage account, and copy them to another storage account one by one by AzCopy .

https://learn.microsoft.com/en-us/azure/storage/scripts/storage-common-transfer-between-storage-accounts

To copyeverything

Copy all containers, directories, and blobs to another storage account by using the azcopy copy command.

azcopy copy 'https://<source-storage-account-name>.blob.core.windows.net/<SAS-token>' 'https://<destination-storage-account-name>.blob.core.windows.net/' --recursive

azcopy copy 'https://mysourceaccount.blob.core.windows.net/?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-07-04T05:30:08Z&st=2019-07-03T21:30:08Z&spr=https&sig=CAfhgnc9gdGktvB=ska7bAiqIddM845yiyFwdMH481QA8%3D' 'https://mydestinationaccount.blob.core.windows.net' --recursive

source

Hendrix

If you have more containers and/or blobs then can be managed manually this is the best answer. AzCopy will also batch the file copies in parallel unlike the other PowerShell script answers which resulted in a 10x performance improvement in my case. Also, if you don't know, (as I didn't), you can easily obtain the needed SAS-tokens from the Azure portal. – Norahnorbert

Answer 3

I used Mick's script in a scheduled Azure Automation Runbook job to backup blob storage.

I made this minor change to it to check if blob already exists in the destination container before attempting copying over:

#Do the copy of everything
foreach ($Blob in $Blobs)
{
   $BlobName = $Blob.Name
   $blob = Get-AzureStorageBlob -Blob $BlobName -Container $ContainerName -Context 
   $DestStorageContext -ErrorAction Ignore
   if (-not $blob)
   {
      Write-Output "Copying $BlobName from $ContainerName"
      $BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
      $BlobCpyAry += $BlobCopy
   }  
} 

Quadrate

You should actually show completed code with your changes. That's more helpful to people. – Nissa

Answer 4

I highly recommend you take a look at Azure Storage Explorer https://azure.microsoft.com/en-us/features/storage-explorer/. It's a lot easier than azcopy, and you can even copy and paste between containers.

Unwished

This is the best way, internally it uses Az copy only but if u have tooling why you even need a script which may lead you to mundane errors :P – Bailiwick

Answer 5

Similar to above but using Bash with Azure CLI and AZCopy - Code is on Github and associated video on YouTube.

https://github.com/J0hnniemac/yt-blobsync

#!/bin/bash
cd /home
app_id=""
tenant=""
sourceurl="https://<>.blob.core.windows.net"
destinationurl="https://<>.blob.core.windows.net"

pemfile="/home/service-principal.pem"

sourceaccount=$(echo $sourceurl | awk -F/ '{print $3}' | awk -F. '{print $1}')
destinationaccount=$(echo $destinationurl | awk -F/ '{print $3}' | awk -F. '{print $1}')

echo $app_id
echo $tenant
echo $sourceurl
echo $destinationurl

echo $sourceaccount
echo $destinationaccount

az login --service-principal --password $pemfile --username $app_id --tenant $tenant

# list storage containers
az storage container list --auth-mode login --account-name $sourceaccount -o=table | awk 'NR>1 {print $1}' | grep networking-guru > src.txt
az storage container list --auth-mode login --account-name $destinationaccount -o=table | awk 'NR>1 {print $1}' | grep networking-guru > dst.txt

grep -vf dst.txt src.txt > diff.txt 

for blob_container in $(cat diff.txt);
        do
        echo $blob_container;
        newcmd="az storage container create --auth-mode login --account-name $destinationaccount -n $blob_container --fail-on-exist" 
        echo "---------------------------------"
        echo $newcmd
        eval $newcmd
done

echo "performing AZCOPY login"
azcopy login --service-principal --certificate-path $pemfile --application-id $app_id --tenant-id $tenant



echo "performing AZCOPY sync for each container"
for blob_container in $(cat src.txt);
   do
    #Create timestame + 30 Minutes for SAS token
    end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`
    sourcesas=`az storage container generate-sas --account-name $sourceaccount --as-user --auth-mode login --name $blob_container --expiry $end --permissions acdlrw`
    echo $sourcesas
    # remove leading and trailing quotes from SAS Token
    sourcesas=$(eval echo $sourcesas)
    echo $sourcesas
    src="$sourceurl/$blob_container?$sourcesas"
    dst="$destinationurl/$blob_container"
    echo $src
    echo $dst
    synccmd="azcopy sync \"$src\" \"$dst\" --recursive --delete-destination=true"
    echo $synccmd
    eval $synccmd
done

Bakery

Answer 6

This is from 2020 and the credit goes to @Mick. I just removed the separate blob copy. And it overwrites the whole container in one line. If the code executes this is guaranteed to work. So no check is given. NB: This will not copy folders and instead give error

#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey

$Containers = Get-AzureStorageContainer -Context $SourceStorageContext

    foreach($Container in $Containers)
{
    $ContainerName = $Container.Name
    if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
    {   
        Write-Output "Creating new container $ContainerName"
        New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
    }
    Write-Output "start copy $ContainerName"
    az storage blob copy start-batch --destination-container "$ContainerName" --account-name "$DestStorageAccount" --account-key "$DestStorageKey" --source-account-name "$SourceStorageAccount" --source-account-key "$SourceStorageKey" --source-container "$ContainerName"
}

Fortyish

Answer 7

Powershell Script:

Select-AzSubscription -Subscription "your subscription name"  # Set subscription id

# Variables
$SourceStorageAccount = "SourceStorageAccount"      # Replace with required Source Storage Account
$DestStorageAccount = "DestStorageAccount"  # Replace with required Destination Storage Account
$SourceResourceGroupName = "SourceResourceGroupName"    # Replace with required source resource group name
$DestResourceGroupName = "DestResourceGroupName"        # Replace with required destination resource group name

Get the storage keys for both the source and destination storage accounts

$SourceStorageKey = Get-AzStorageAccountKey -Name $SourceStorageAccount -ResourceGroupName $SourceResourceGroupName
$DestStorageKey = Get-AzStorageAccountKey -Name $DestStorageAccount -ResourceGroupName $DestResourceGroupName

$SourceStorageContext = New-AzStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey.Value[0]
$DestStorageContext = New-AzStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey.Value[0]

Loop for each container

$Containers = Get-AzStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{

    $ContainerName = $Container.Name
    if ($ContainerName -eq "snow-cmdb"){
        $ContainerName = $Container.Name+"copy3"
    if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
    {   
        Write-Output "Creating new container $ContainerName"
        New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
    }

    $Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
    $BlobCpyAry = @() #Create array of objects

    #Copy every thing
    foreach ($Blob in $Blobs)
    {
       $BlobName = $Blob.Name
       Write-Output "Copying $BlobName from $ContainerName"
       $BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
       $BlobCpyAry += $BlobCopy
    }

    #Check Status
    foreach ($BlobCopy in $BlobCpyAry)
    {
       #Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState output with % copied
       $CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
       $Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100) 
       Write-Output $Message
    }
    }
}

Drouin

Answer 8

Did you see this tool?: https://azure.microsoft.com/en-us/documentation/articles/storage-use-azcopy/, it may copy all blobs from one container to another.

Though, when I needed to move content between accounts, I ended up with downloading all content to VM and then uploading it back to new account.

Pectize

I wanted to copy the 100 containers that I have from one account to other. Copying the blocks can be done with many tools available, wanted to know how the container themself can be copied. – Eraser

@CourageousHeart, containers can't be copied. You'll have to create destination containers by yourself: either by hands or programmatically. Tools can only iterate through blobs within container and copy them one-by-one. – Pectize

Currently I have some 100 containers, not sure whether right thing to create them manually. surprised to know there are no tools that can copy the containers as well :( – Eraser

You can create a powershell script, use Azure storage PSH cmdlets to iterate all the containers, and use PSH cmdlets to call the AzCopy exe to copy each container. – Mauney