Why would AzureSearch SDK spontaneously fail with transport connection issue?

Question

We had some big excitement yesterday. Two of our web apps (both deployed and untouched for at least 3 months) that use AzureSearch's SDK stopped working at different times (one early morning; the other in the evening). The indexClient.Documents.Search method started failing with this error:

HTTPS handshake to XXXXsearch.search.windows.net (for #435) failed. System.IO.IOException Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. < An existing connection was forcibly closed by the remote host

After a mad google scramble we added these 3 lines just before the search which fixed it.

            const SslProtocols _Tls12 = (SslProtocols)0x00000C00;
            const SecurityProtocolType Tls12 = (SecurityProtocolType)_Tls12;
            ServicePointManager.SecurityProtocol = Tls12;

All I can imagine is we missed some kind of "AzureSDK is changing" email? This was very bad and we got lucky on a quick solution or it could have been a disaster. Does anyone know why this would have happened?

Subsist

This has happened to us today! Our site is currently down because of this. – Supra

Your 3 lines have fixed the issue for me also! Thank you!!! However, not before I'd upgraded our use of Microsoft.Azure.Search from 0.12.preview to 10.1.0 - yes, this service has run without interruption since 2015! – Supra

@Supra happy to help! It was one of my devs who found the fix - even then I couldn't believe we needed a CODE FIX! The timing was incredibly bad as I had just deployed a new app that shared some resources of one of these apps. I was sure I'd done something stupid. I was relieved it wasn't my fault but still so stressful and customer was furious. – Subsist

Hi, the same happened with our application and it suddenly stopped working in production. I have added the same fix as shown by you above and it resolves the issue in my local env. The problem is this code is still not working for us on test/prod env. It breaks with the same TLS error. we have verified that our servers have TLS 1.2 enabled. Is there anything else we need to check here? – Goatherd

Answer 1

I am a Program Manager in the Azure Cognitive Search engineering team. First off, I am sorry to hear that you and others had client connectivity issues due to the latest service update that removed support for TLS 1.0 and 1.1. These types of changes are rare and we strive to never break customer code in any update. In this case, it was critical to help protect the privacy of information communicated over the Internet. TLS 1.2 is a standard that provides security improvements over previous versions. You can learn more about this here.

More information on how to resolve this issue can be found here. We strongly recommend that you avoid hard coding the TLS version (as mentioned above), as TLS 1.2 will eventually be replaced by the newest released standard TLS 1.3 which is faster and has improved security. It is better to let .NET or the OS choose the TLS version for you but that is discussed more in the above links.

We had sent a notification on this upcoming change in the portal in early January, but we realize that it is easy to miss these and apologize for the issues it has subsequently caused. Please know we want to do what we can to help. Since we are asked not to use StackOverflow to help resolve individual support cases, if you would like individual help regarding your service, we would ask you to open a support case.

Alphorn

Answer 2

The transport connection errors are the result of a known issue in Azure Search.

And occasionally there are unexpected connectivity issues. Try running the document through your indexer again later. For more details, you could refer to this article.

Lunar

This was not an intermittent issue. This was consistently failing for > 1 hour while we were scrambling for a fix. – Subsist