Azure CloudBlobContainer.CreateIfNotExists returns 403 forbidden

Question

I am calling CloudBlobContainer.CreateIfNotExist (see FindOrCreatePrivateBlobContainer method below) indirectly from within a Web API service, but it returns the following 403 forbidden error message:

<Error>
<Message>An error has occurred.</Message>
<ExceptionMessage>
The remote server returned an error: (403) Forbidden.
</ExceptionMessage>
<ExceptionType>Microsoft.WindowsAzure.Storage.StorageException</ExceptionType>
<StackTrace>
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.Exists(Boolean primaryOnly, BlobRequestOptions requestOptions, OperationContext operationContext) at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExists(BlobContainerPublicAccessType accessType, BlobRequestOptions requestOptions, OperationContext operationContext) at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExists(BlobRequestOptions requestOptions, OperationContext operationContext) at [Obfuscated].DocumentManagement.BlobStorage.BlobHelper.FindOrCreatePrivateBlobContainer(String ContainerName, String AccountConnectionString) in c:\Users\[Obfuscated]\Desktop\[ProjectNameObfuscated]Online\[Obfuscated].DocumentManagement.BlobStorage\BlobHelper.cs:line 25 at [Obfuscated].DocumentManagement.BlobStorage.BlobFileItemHandler.GetStream(Int64 FileItemId) in c:\Users\[Obfuscated]\Desktop\[ProjectNameObfuscated]Online\[Obfuscated].DocumentManagement.BlobStorage\BlobFileItemHandler.cs:line 114 at [Obfuscated].DocumentManagement.Service.Controllers.FileItemController.Get(String ServiceAuthKey, Int64 FileItemId) in c:\Users\[Obfuscated]\Desktop\[ProjectNameObfuscated]Online\[Obfuscated].DocumentManagement.Service\Controllers\FileItemController.cs:line 148 at lambda_method(Closure , Object , Object[] ) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
</StackTrace>
<InnerException>
<Message>An error has occurred.</Message>
<ExceptionMessage>
The remote server returned an error: (403) Forbidden.
</ExceptionMessage>
<ExceptionType>System.Net.WebException</ExceptionType>
<StackTrace>
at System.Net.HttpWebRequest.GetResponse() at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
</StackTrace>
</InnerException>
</Error>

Here is the code that is generating the error:

     public HttpResponseMessage Get(string ServiceAuthKey, Int64 FileItemId)
        {
            if (!CheckServiceAuthKey(ServiceAuthKey).IsSuccessStatusCode)
                return new HttpResponseMessage(HttpStatusCode.Unauthorized);

            HttpRequest request = HttpContext.Current.Request;

            FileItem fi = null;
            using (DocumentDbContext db = new DocumentDbContext())
            {
                fi = db.FileItems.Find(FileItemId);
            }


            BlobFileItemHandler fih = new BlobFileItemHandler();
            Stream s = fih.GetStream(FileItemId);


            // -------- DOWNLOAD FILE TO CLIENT -------- 

            HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
            response.Content = new StreamContent(s);
            //a text file is actually an octet-stream (pdf, etc)
            response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
            //we used attachment to force download
            response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
            response.Content.Headers.ContentDisposition.FileName = fi.PublicFileName;


            return response;
        }

 public Stream GetStream(Int64 FileItemId)
        {
            CloudBlobContainer c = BlobHelper.FindOrCreatePrivateBlobContainer("[Obfuscated]-dms", AccountConnectionString);

            using (DocumentDbContext db = new DocumentDbContext())
            {
                FileItem fi = db.FileItems.Find(FileItemId);

                CloudBlockBlob blob = c.GetDirectoryReference(fi.FilePathOnServer).GetBlockBlobReference(fi.PrivateFileName);
                bool blobExists = blob.Exists();

                if (!blobExists)
                    throw new System.IO.FileNotFoundException();

                Stream stream = new MemoryStream();
                blob.DownloadToStream(stream);
                long streamlen = stream.Length;
                stream.Position = 0;
                return stream;
            }

        }

public static CloudBlobContainer FindOrCreatePrivateBlobContainer(string ContainerName, string AccountConnectionString)
        {
            Trace.TraceInformation("FindOrCreatePrivateBlobContainer '" + ContainerName + "' with connectionstring '" + AccountConnectionString + "'");
            CloudStorageAccount account = CloudStorageAccount.Parse(AccountConnectionString);
            CloudBlobClient blobClient = account.CreateCloudBlobClient();
            CloudBlobContainer container = blobClient.GetContainerReference(ContainerName);
            container.CreateIfNotExists();
            return container;
        }

I need some help as to troubleshoot the cause of this error. I have tried the following:

1. Checked that the name of the to-be-created container is valid and in this specific cases consists of only lowercase letters (no special or uppercase characters).
2. I have read that differences in time zones between the Azure servers and the calling servers can lead to 403 forbidden error messages. This error occurs irrespective of whether I am running the service from my personal computer (with timezone set to UTC) or running from an Azure deployment.
3. I have checked my connection string and account key, which seems to be correct. It is in the following format: <add key="MyStuff.DocumentManagement.ConnectionString" value="DefaultEndpointsProtocol=http;AccountName=MyStuffAccount;AccountKey=[obfuscated]" />
4. I have tried switching between http and https, with no difference in result.
5. I can confirm I am running against the latest version of the Azure storage API (4.1.0)
6. I am able to connect to Azure storage and create a new container via VS 2013 server explorer

Please help!

UPDATE

Here is the output of the error after I enabled tracing: Application: 2014-07-13T19:08:03 PID[6888] Error
Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden. Application: at System.Net.HttpWebRequest.GetResponse() Application: at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext) Application: --- End of inner exception stack trace --- Application: at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext) Application: at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.Exists(Boolean primaryOnly, BlobRequestOptions requestOptions, OperationContext operationContext) Application: at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExists(BlobContainerPublicAccessType accessType, BlobRequestOptions requestOptions, OperationContext operationContext) Application: at [Obfuscated].DocumentManagement.BlobStorage.BlobHelper.FindOrCreatePrivateBlobContainer(String ContainerName, String AccountConnectionString) Application: at [Obfuscated].DocumentManagement.BlobStorage.BlobFileItemHandler.GetStream(Int64 FileItemId) Application: Request Information Application: RequestID:fce980ad-a673-4ef1-b55d-d017a49845c8 Application: RequestDate:Sun, 13 Jul 2014 19:08:02 GMT Application: StatusMessage:Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Predigest

Were you able to access the storage account from VS 2013 explorer using the same key used in code above? If not, can you login from portal and confirm that your keys have not ben re-generated. Otherwise, to help debug this further, can you run fiddler and share a fiddler trace of your request when the failure happens. – Toyatoyama

@VinayShah-Microsoft Thanks, that was actually exactly the issue - however I have fixed that a couple of days ago (just have not had time to update the answer here). Does this key generation occur automatically? – Predigest

Good to know that your issue is resolved. Key re-generation should not happen automatically. It needs to be manually invoked through the portal or service management api. – Toyatoyama

@VinayShah-Microsoft Thanks. Weird things tend to happen when you are still working at 2 AM in the morning. – Predigest

Answer 1

As we discussed in comments above, please make sure your keys valid when you start getting 403 status code back from the service. Keys can change if they have been re-generated through the portal or using service management API's

Toyatoyama

Answer 2

This may not be the solution to this particular problem, but it might help someone else.

I was getting a 403 error and had a hard time tracking down the solution. I finally figured out that my development machine's time was off by 2 hours. When I set the time correctly, the 403 went away.

Azure requires the UTC timestamp to be within 15 minutes of the request time.

Femur

This has also happened to me - where there was a problem with my web server's international datetime server synching. I applied a tool called Atomic Clock sync to synchronise more often with an internation datetime server to solve this issue. Just thought I would put this here as I believe some people with this problem might end up here. – Predigest

Answer 3

If you have recently upgraded your WindowsAzure.Storage instead of using the deprecated WindowsAzure.StorageClient dll then you could be experiencing this issue as the BlobRequestOptions and OperationContext have not been set

MSDN Link with CloudBlobContainer Create If Not Exists Method

http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.cloudblobcontainer.createifnotexists.aspx

Charade

Answer 4

For me, it turned out that we had an IP restriction on the storage account which was causing a 403 Forbidden. The way to fix it was to access the Azure Portal and go to:

Storage Accounts → "StorageAccountName" → Firewalls and Virtual Networks

And then make sure that your IP is in the allowed section, or check "Allow access from All Networks".

Fidgety

Answer 5

I had a similar issue when trying to connect to my local azure storage emulator. Through Azure Storage explorer the connection was successful but through a custom tool which uses REST API it was throwing 403 error. I had to manually include endpoints in the config and use http instead https.

More on that here (https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string)

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;
AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;
BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;
TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;
QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;

Telling

Answer 6

Try the following: 1. Generate new access keys to use in your App.config file

OR 2. Login into your azure portal Under storage account Select ->Firewalls and virtual networks --> Under the blade, Enable Allow access from "All Networks" Option

I hope this works for someone!

enter image description here

Marola

Answer 7

I saw the same error message when calling

Microsoft.WindowsAzure.Storage.Blob.CloudBlobClient.GetBlobReferenceFromServerAsync(Uri blobUri)

accidentally using a blobUri value that pointed to a different URL than the CloudBlobClient.BaseUri was pointing to.

Dublin