Azure Pipelines local agent failing to connect with SSL error

Question

We have an on premise server (Windows Server 2012 R2) with an Azure Pipelines agent running on it. Today (31st Jan 2022) this agent could not longer connect to our Azure DevOps organisation.

Judging by the log files, I assume this is because it is trying to connect with an older TLS version, which as of today is no longer available - https://devblogs.microsoft.com/devops/azure-devops-services-to-require-tls-1-2/

So I followed the instructions on how to make sure TLS 1.2 was enabled, and confirmed my settings in the registry editor and by running the PowerShell script suggested here - https://learn.microsoft.com/en-us/security/engineering/solving-tls1-problem#update-windows-powershell-scripts-or-related-registry-settings

All seems ok, yet it still fails to connect with the same issue. The machine has been restarted as well. If I try the URL it is requesting in the in built Internet Explorer browser, it fails, but with Chrome it succeeds, so it must still be trying to connect with TLS 1.2, but I don't know why. I've tried reinstalling the agent (with the latest build) as well but it fails on the same error. Any suggestions?

Blues

I have the same problem, in a 2012 R2 server, other servers are working fine. Tried to change anything related in registry I was able to find, and installed .net framework too -- but wasn't able to figure out what's wrong. – Heine

If you understand more about this than me, that probably helps: developercommunity.visualstudio.com/t/… – Heine

Answer 1

Enabling below Cyphers with IISCrypto on the server helped us fix the issue

Cipher Suites

TLS 1.2 (suites in server-preferred order)

• TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x9f) DH 2048 bits FS 256
• TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 2048 bits FS 128

This from Vijay's solution

Shana

Good find, I didn't find that when I searched! – Blues

Note that you must reboot the server for the settings to be applied. IIS Crypto includes a /reboot option. If you use Group Policy, you have to force the policy update, then reboot. – Addle

Answer 2

On the affected computer you can try to run an analysis with AzDO TLS 1.2-transition readiness checker: https://github.com/microsoft/azure-devops-tls12/blob/main/AzureDevOpsTls12Analysis.ps1

The script checks if the client TLS 1.2 usage is enabled, checks cipher suite issues and checks presence of switches needed to make legacy .NET Framework apps favor TLS 1.2. Based on findings it prints mitigation suggestions or generates tailored mitigation script (ps1).

Simulating the issue described in the question, the script gives suggestion:

MITIGATION 'gpeditREM': via Local Group Policy setting
    Run gpedit.msc:
    - Navigate to "Computer Config/Administrative Templates/Network/SSL Config Settings"
    - Choose setting "SSL Cipher Suite Order" -> Edit
    - If 'Enabled' is not checked, then continue to the next mitigation below.
    - If 'Enabled' is checked:
      - *either* change to 'Not configured' (resets to OS-default setting)"
      - *or* keep 'Enabled' and in field 'SSL Cipher Suites' add at least one of the following items to comma-separated list:
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
    - Press 'OK' button
    Restart the computer

MITIGATION 'regFunctionsDEL': deletion of cipher suite list in registry
    Mitigation script generated at .\Mitigation-regFunctionsDEL.ps1
    Run the mitigation script as Administrator:
    - If 'Done!' is printed, then operation was successfull.
    - If 'Mitigation was not effective!' is printed, then cipher suites are enforced via domain GPO - talk to your domain administrator.

Screenshot: TLS 1.2-transition readiness checker on WS 2012 R2

Windows Server 2012 R2 (and older) are vulnerable to cipher suite problems because it supports only two out of four TLS 1.2 cipher suites currently required by AzDevOps (even those two are supported only if this update is applied).

Keep in mind Azure DevOps rolled-back its January 31st TLS 1.0/1.1 deprecation (except for IPv6 endpoints). Full deprecation is currently scheduled not sooner than on March 31st 2022 (announcement)

• Update (May 2022): Azure DevOps team has completed TLS 1.0/1.1 deprecation on dev.azure.com domain. Subdomains *.visualstudio.com will follow.
• Update (August 2022): Azure DevOps has started TLS 1.0/1.1 deprecation on *.visualstudio.com.
• Update (October 2022): Azure DevOps declared TLS 1.0/1.1 deprecation finished throughout all domains.
• Update (January 2023): Azure DevOps endpoints changed list of cipher suites they support. Removed TLS_DHE_* ciphers, added two ECDHE+CBC ciphers. This may cause issues on Win Server 2012 R2 machines which have ECDHE+CBC disabled. <-- this has been reverted on February 02 2023

Frum

Thanks. The tool revealed two missing ciphers for us. Added, restarted and everything worked. – Dockage

Answer 3

We faced this issue with Azure DevOps unable to publish to Azure App Services due to the TLS 1.2 mandate. We have self-hosted agents, and despite the boxes showing having .NET 4.8 installed, we ran the below registry edits in an elevated command prompt per this doc which restored connections:

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 /v SystemDefaultTlsVersions /t REG_DWORD /d 1 /f /reg:64

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 /v SystemDefaultTlsVersions /t REG_DWORD /d 1 /f /reg:32

Dialyse

Answer 4

The same thing happened to me, but I was able to solve it by installing this application on the server, which is used to activate the TLS2 protocols with just one click and the SSL Cipher Suites:

https://www.nartac.com/Products/IISCrypto/

TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x9f) DH 2048 bits FS 256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 2048 bits FS 128

Collectanea

Are you affiliated with Nartac? – Shiite

Welcome to Stack Overflow! When linking to your own site or content (or content that you are affiliated with), you must disclose your affiliation in the answer in order for it not to be considered spam. – Lacefield