Windows Azure Storage Certificate Expired

Question

The certificate for our Azure blob storage expired today. This is not a certificate provided by us but provided by Microsoft as show in the picture below. How does one go about fixing this? I have tried searching for a solution but found nothing. Our app cannot connect to the storage as the certificate has expired and we are getting an error indicating: Could not establish trust relationship for the SSL/TLS secure channel

Marcelinomarcell

Wow, thanks to your post I realized that I am not able to use some Azure Blob browsing tools such as CloudBerry (SSL issue: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.). – Potpourri

I opened a support case with Microsoft. This affects the *.blob.core.windows.net SSL cert, which in turn affects all Blob Storage accounts Worldwide. Someone at Microsoft should lose their job for this. – Verse

The problem is being investigated seriously and dashboard is updated with the current status. Please keep an eye on dashboard for frequent updates: windowsazure.com/en-us/support/service-dashboard – Calendula

Scott Hanselman is working on it now, you can track it on Twitter: twitter.com/search/realtime?q=azure+blob&src=typd – Ethel

Moderator Note: Please keep the noise on this post to a minimum. You know about it, I know about it, and now Microsoft knows about it. We all agree it's a travesty; save your editorial comments for Twitter. There's a temporary workaround already posted below. – Dorkas

Answer 1

As a temporary measure I was able to log into the azure portal and change the protocol part of the connection string from https to http.

Sholokhov

Keep blobs containing sensitive data accessible only over HTTPS. – Berbera

How do you change the protocol? I don't see it in the portal. – Ethel

On the configure tab scroll down to settings, find your connection string and change DefaultEndpointsProtocol=https; to DefaultEndpointsProtocol=http; – Sholokhov

Be careful. If you're changing it in portal, your previously deployed package may not reload because it is stored in Azure storage (I believe). Doing this caused the first 2 worker roles in my pool to become unresponsive and continuously recycle. – Unvoice

If you do this, be sure to leave the connection string for Diagnostics running with HTTPS. If you change it to HTTP, the monitor throws an exception on startup that will cause your roles to recycle endlessly. – Hau

Answer 2

Two more possible solutions if you can RDP into your roles.

1. Change the configuration manually in the c:\Config directory.
2. Build a DLL that's patched to work around the problem, and manually upload it via RDP. The workaround could be hardcoded connection strings, or put in code to accept expired certs. For example: ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };

(Hat tips to AlexNS on MSDN forums for idea #2 and to Jason Vallery for the cert validation callback code)

As noted in the comments, disabling HTTPS and/or ignoring certificate validation errors can compromise the security of your communications. So think hard before you do this, and change it back as soon as Microsoft fixes this problem.

Hau

Just be warned to all that this could effectively enable MITM type attacks. So if you do this, do this knowing the risk you expose yourself to and revert it the second Microsoft fixes this!! – Chops

Answer 3

We were able to dodge most of this in the first place through explicit use of HTTP endpoints for storage (we don't store anything too sensitive there).

In case you're in a similar situation and can do with HTTP endpoints, there is a workaround that allows you to upgrade your roles permanently. It involves Azure Powershell deployments with local packages and seems to work even when upgrades via the both portals continue to fail.

Diastasis

Answer 4

Just as a note - if you switch to http from https then the transfer mechanism no longer makes sure the data is transferred correctly, and you may need to check the MD5 of the blob.

StorageClient < 2.0 manages this sometimes with uploads, but reading this article, never from downloads.

For StorageClient 2.0, you may need to change the BlobRequestOptions to UseTransactionalMD5 (as detailed here)

Allpurpose