AZCopy: Set the file Content-Type

Question

We are trying to use AZCopy in our deployment script to upload some assets directly into our storage which is exposed via a CDN on Azure.

When we upload the files, the Content-Type is application/octet-stream but we would need to be able to specify by example text/javascript

Is there a way to achieve this? We don't bother having to call AZCopy several times for each file types.

Discoverer

Answer 1

The latest version of AzCopy (v3.1.0 and v4.1.0-preview) has included the support for setting content type, please download and find more details at http://aka.ms/azcopy.

To be more specific, you can set the content type via the option /SetContentType:[content-type].

AzCopy /Source:D:\test\ /Dest:https://myaccount.blob.core.windows.net/myContainer/ /DestKey:key /Pattern:ab /SetContentType

AzCopy /Source:D:\test\ /Dest:https://myaccount.blob.core.windows.net/myContainer/ /DestKey:key /Pattern:ab /SetContentType:video/mp4

If "Content-Type" is not specified at the /SetContentType option, AzCopy will set each blob’s content type according to its file extension. To set the same content type for all the blobs, you must explicitly specify a value for “Content-Type", for example, /SetContentType:video/mp4. Note that this option is only applicable when uploading blobs to the storage end points.

Jurdi

Answer 2

I found a way of doing this using Azure Pipelines:

- task: AzureFileCopy@4
  displayName: "Azure Storage - copy new files"
  inputs:
    SourcePath: '$(System.DefaultWorkingDirectory)/dist/*'
    azureSubscription: 'johnykes-PAYG(xxxxx-b455-4c4d-a8f8-c2a5fd479f10)'
    Destination: 'AzureBlob'
    storage: 'johnykeschatfrontend'
    ContainerName: '$web'

- task: AzureFileCopy@4
  displayName: "Azure Storage - overwrite .js files with correct Content Type"
  inputs:
    SourcePath: '$(System.DefaultWorkingDirectory)/dist/*.js'
    azureSubscription: 'johnykes-PAYG(xxxxx-b455-4c4d-a8f8-c2a5fd479f10)'
    Destination: 'AzureBlob'
    storage: 'johnykeschatfrontend'
    ContainerName: '$web'
    AdditionalArgumentsForBlobCopy: '--content-type "application/javascript"'

Please let me know if you find a way of doing this using the Azure CLI or in my way but recursively (for all .js files).

Helix

there was possibility to do it with az copy task peterrombouts.nl/2017/08/11/… In my project it worked with version 1 – Kew

Answer 3

To add to the other answers here for anyone still finding this question, there is (now) an AzCopyConfig.json file where you can specify the MIME types for each file extension.

AzCopy determines content type of a blob based on a JSON file that stores content type to file extension mapping. This JSON file is named AzCopyConfig.json, and is located in the AzCopy directory. If you have a file type that is not in the list you can append the mapping to the JSON file:

{ "MIMETypeMapping": { ".myext": "text/mycustomtype", . . } }

Source: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy#automatically-determine-content-type-of-a-blob

Decedent

Link no longer points to the right section, and I can't find it anymore. Is this still possible? – Cuisine

Answer 4

Looking at the options available in AZCopy, I don't think it is directly possible. You would need to write some code to set the content type of the blob once it has been uploaded.

What you could do is fetch the list of blobs in the desired container. Blob listing would give you the current content type of each blob. Then you would need to loop over this list, find out the content type from the file extension (Get MIME type from filename extension) and then set the blob's properties with updated content type.

Gregorygregrory

Thanks Gaurav, we thought about a similar solution but were hoping an easier way with AZCopy, but I guess we will do as you suggested! – Discoverer

Answer 5

UPDATE: Note that my issue was a bit different: i needed to rename the files in a blob when downloading them with AzCopy because they had no extension. Probably this script can be reversed to achieve your goal!

I could not find a way to make that work using azcopy, so I realized a powershell script that connects to my storage, downloads the items inside a 'Download' folder (located in the same script location) and renames them by adding the extension.

Note that this scenario was easy to handle as I knew that every file only required to add the .pdf extension. I will try to improve this script and include a better solution to handle different data types if I can figure it out!

1-You must download Powershell 7.*

2-Install the Azure Modules for Powershell by changing the ExecutionPolicy for CurrentUser then install the module (it may take a while, you can add the -Verbose parameter to have more info:

• Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
• Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force

3-Customize this script and save it with .ps1 extension:

 
## Input Parameters  
param  ($subscriptionId="", ## Your subscriptionId here, if not specified "Connect-AzAccount" will use the first available subscription.
        $resourceGroupName="your-resource-name",  
        $storageAccName="your-storage-account-name",
        $storageContainer="your-storage-container",
        $itemsCount=-1) ## Amount of items to download. If -1 (default value) or 0 it will download everything inside the container.

$downloadPath=".\Download"
$downloadLocation="Download"
 
## Connect to Azure Account and set the Context
if ($subscriptionId -ne "")
{
    Connect-AzAccount -Subscription $subscriptionId
}
else
{
    Connect-AzAccount
}
 
 
Function BlobCycleAndRename
{
    param([array]$Contents) 
    
    foreach($content in $Contents)
    {  
        $itemName=$content.Name
        ## Download the blob content  
        Get-AzStorageBlobContent -Container $container.Name  -Context $ctx -Blob $itemName -Destination $destination -Force  
        
        ## Rename the items, if an item with the same name already exists, it is removed first.
        if (Test-Path ($destination+"\"+($itemName+".your-extension-here")))
        {
            Remove-Item -Path ($destination+"\"+($itemName+".your-extension-here"))
        }
        
        Rename-Item -Path ($destination+"\"+$itemName) -NewName ($itemName+".your-extension-here") -Force
    }  
    
}
 
## Function to download all blob contents  
Function DownloadBlobContents  
{  
    Write-Host -ForegroundColor Green "Download blob contents from storage container.."    
        
    ## Get the storage account  
    $storageAcc=Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccName     
    ## Get the storage account context  
    $ctx=$storageAcc.Context  
    ## Get the containers 
    $container=Get-AzStorageContainer -Context $ctx -Name $storageContainer

    ## check if folder exists  
    $folderPath=$downloadPath+"\"+$container.Name  
    $destination=$downloadLocation+"\"+$container.Name  
    $folderExists=Test-Path -Path $folderPath  
    if($folderExists)  
    {  
        Write-Host -ForegroundColor Magenta $container.Name "- folder exists"  
        ## Get the blob contents from the container  
        $blobContent
        
        if ($itemsCount -le 0) 
        {
            $blobContents=Get-AzStorageBlob -Container $container.Name  -Context $ctx
            
        }
        else
        {
            $blobContents=Get-AzStorageBlob -Container $container.Name  -Context $ctx -MaxCount $itemsCount
        }

        ## Download and rename blobs
        BlobCycleAndRename -Contents $blobContents 
         
    }  
    else  
    {        
        Write-Host -ForegroundColor Magenta $container.Name "- folder does not exist"  
        ## Create the new folder  
        New-Item -ItemType Directory -Path $folderPath  
        
        ## Get the blob contents from the container  
        $blobContents
        if ($itemsCount -le 0) 
        {
            $blobContents=Get-AzStorageBlob -Container $container.Name  -Context $ctx
            
        }
        else
        {
            $blobContents=Get-AzStorageBlob -Container $container.Name  -Context $ctx -MaxCount $itemsCount
        }
            
        ## Download and rename blobs
        BlobCycleAndRename -Contents $blobContents
 
    }     
}   
  
DownloadBlobContents  
 
## Disconnect from Azure Account  
Disconnect-AzAccount  

Some credits go to: https://www.c-sharpcorner.com/blogs/how-to-download-blob-contents-from-azure-storage-account-using-powershell

---

PREVIOUS POST Note that my issue is a bit different (it's "reversed" somehow): I need to set a file extension when downloading files from AzCopy, because they have been uploaded with no extension.

I am answering to @Kolky here, but I have not enough rep to directly add comments!

As pointed out, the MIMETypeMapping page seems to not exist anymore, however I was still able to find something useful.

You have to set the AZCOPY_CONTENT_TYPE_MAP Environment variable and provide it a json config file where you can provide the desired mapping type.

      "MIMETypeMapping": {
        ".323": "text/h323",
        ".aaf": "application/octet-stream",
        ".aca": "application/octet-stream",
        ".accdb": "application/msaccess",
      }

You can access the configured AzCopy Environment variables by using the command AzCopy env (or /path/AzCopy.exe). I am still trying to figure out if I am doing something wrong, as this seems to not be working, but this can still be helpful to someone! :)

A couple of links for reference:

https://github.com/Azure/azure-storage-azcopy/wiki/Custom-mime-mapping

https://learn.microsoft.com/it-it/azure/storage/common/storage-ref-azcopy-configuration-settings#AZCOPY_CONTENT_TYPE_MAP

https://github.com/Azure/azure-storage-azcopy/issues/136

Owings

hi, did it worked? – Kew

Hello! Unfortunately I could not find a way to make this work with AzCopy. My solution was to make a PowerShell script and by using the Azure Modules for Powershell I have downloaded the items and renamed then. I am editing my post to include my solution. Just note that to me that was pretty easy as I only needed to add the '.pdf' extension. I will try to figure out how to handle multiple types and post a more complete solution as soon as possible! Feel free to improve that and share it here, of course! :) – Owings