Get-AzureBlobContent throwing error when run from Azure Automation account

Question

I am receiving a failure while trying to download blob (JSON file) from Azure storage account from my Azure Automation account. It looks like an authorization issue.

This works on my local laptop, but does not work on Azure Automation Account. Does not work even if I make the container "public"

I have assigned OWNER privileges for the Automation accounts's service principle on the Resource Group (Automation account + Storage account stay in this RG) and specifically on the Storage Account as well:

Below is the code:

$connectionName = "AzureRunAsConnection"
try
{
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName         

    "Logging in to Azure..."
    Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 
}
catch {
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    } else{
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

$config_file_resource_group_name = "vg-datalake-manjunath"
$config_file_storage_account_name = "datalakelog"
$primary_key = (Get-AzureRmStorageAccountKey -ResourceGroupName $config_file_resource_group_name -AccountName $config_file_storage_account_name).value[0]
 $config_file_context = New-AzureStorageContext -StorageAccountName $config_file_storage_account_name -StorageAccountKey $primary_key

 Get-AzureStorageBlobContent -Blob "mw_services.json" -Container "fwconfigfiles" -Destination "C:\temp\mw_services.json" -Context $config_file_context

 get-content "C:\temp\mw_services.json" | write-output

ERROR:

Get-AzureStorageBlobContent : The remote server returned an error: (403) Forbidden. HTTP Status Code: 403 - HTTP Error 
Message: This request is not authorized to perform this operation.
At line:30 char:2
+  Get-AzureStorageBlobContent -Blob "mw_services.json" -Container "fwc ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Get-AzureStorageBlobContent], StorageException
    + FullyQualifiedErrorId : 
StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobContentCommand

Gravimetric

Possible duplicate of Azure automation, PowerShell to fetch a file in private blob container – Washroom

This works on my local laptop, but does not work on Azure Automation Account. Does not work even if I make the container "public" – Gravimetric

Please check if the storage account is part of a virtual network or firewalled. That could be the reason for this error. – Thomasenathomasin

@GauravMantri yes. That was the issue. If I select "Allow All Networks", it works. But we will not enable that in our environment. We allow access to the storage account via ProxyRules. However, if I enable "Allow Microsoft services to access this storage account", the code still fails. Any idea? – Gravimetric

Looking here: learn.microsoft.com/en-us/azure/storage/common/…, I believe Azure Automation is not part of trusted Azure Services. Not sure why this is not part of trusted services though :). – Thomasenathomasin

Thanks for the confirmation. Do we have any alternatives? – Gravimetric

I’m not sure. I’ve asked the question on Twitter though. Let’s hope someone from Azure team responds. Will provide an update once I hear back. – Thomasenathomasin

Please see this post on MSDN Forums: social.technet.microsoft.com/Forums/en-US/…. Looks like you will need to look up the IP address for Azure Automation service and add an exception. – Thomasenathomasin

Hope Microsoft address your question. However, regarding the MSDN Forum, it is very difficult to update the Azure IP address and maintain it, because they keep updating the XML file. Also, we have a logic (security implementation) that only "Client proxy" address has to be allowed as a firewall for the PaaS services. (like storage account, ADLA, ADLS etc.,) – Gravimetric

One approach would be to use a Hybrid Runbook worker and integrate the Storage Account into the same Vnet as the Hybrid Runbook worker. This way you don't need access on the internet facing side of the Storage Account. Another approach would be the use of a SAS token. – Canova

We are looking on the Hybrid Worker solution. Meanwhile, regarding "whitelisting Azure Datacenter IPs", the WestEurope region has 123 ranges, however, one Storage Account can have maximum of 100 Network Rules. So this solution is not working. Any workaround ? Each storage account supports up to 100 IP network rules, which may be combined with Virtual network rules. learn.microsoft.com/en-us/azure/storage/common/… – Gravimetric

Answer 1

The possible reason is that you may configure to selected networks to access.

If you enable this option, and whether you tick "allow trusted microsoft services to access", you would get this error, since automation is not listed under MS trusted services. see https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security#trusted-microsoft-services.

Eternalize

The cmdlet "Get-AzureStorageBlobContent" does not work even if we click on "Allow trusted Microsoft services", because "Azure Automation" service is not under "trusted service". – Gravimetric

I have tested the cmdlet "Get-AzureStorageBlobContent, and it works well. I use the code below. Set-AzureRmCurrentStorageAccount –ResourceGroupname "" –StorageAccountName "" $Container = Get-AzureStorageContainer -Container "" Get-AzureStorageBlobContent -Container $Container.Name -Blob "1.txt" -Destination "C:\1.txt" get-content "C:\1.txt" | write-output – Eternalize

1. From the Storage Account's firewall - Select "Selected Networks" 2. Check "Allow trusted Microsoft services to access this storage account" 3. Run your code from an Azure automation runbook. The code still fails with the error ==> Get-AzureStorageContainer : The remote server returned an error: (403) Forbidden. HTTP Status Code: 403 - HTTP Error Message: This request is not authorized to perform this operation. – Gravimetric

Sorry, my mistake. The azure automation is not in MS trusted service based on the document here learn.microsoft.com/en-us/azure/storage/common/…. So, to access storage from azure automation, you need to set "allow access from all networks". Let me correct my answer. Thanks. – Eternalize