Azure Hosted Service Bus : "The X.509 certificate CN=servicebus.windows.net is not in the trusted people store."

Question

Using Azure SDK 2.3 on my vs2013 development VM I can consume Service Bus queues hosted in Azure painlessly. However, on Windows Server 2008 R2 Standard SP1, it looks like Windows can not trust the involved certificates and an exception is thrown.

The line that throws :

// Send the message
await queueclient.SendAsync(message);

Exception message :

The X.509 certificate CN=servicebus.windows.net is not in the trusted people store. The X.509 certificate CN=servicebus.windows.net chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. A certificate chain could not be built to a trusted root authority.

The CAPI2 logs (attached below) pointed to a trust issue so I compared certificates installed on both machines. The following certificates are absent on the server :

Intermediate Certification Authorities > Microsoft Internet Authority (Issued by Baltimore CyberTrust Root)

Intermediate Certification Authorities > MSIT Machine Auth CA 2 (Issued by Microsoft Internet Authority)

The questions :

1. Where does the certificates come from?
2. Why are they missing from the server?
3. How to fix this issue?

Possible trails (updated) :

1. Install Azure SDK 2.3 for Visual Studio 2013 on the server
2. Install all Windows Updates on the server

I tried :

<appSettings>
  <add key="Microsoft.ServiceBus.X509RevocationMode" value="NoCheck"/>
</appSettings>

---

CAPI2 Verify Chain Policy event :

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-CAPI2" Guid="{5bbca4a8-b209-48dc-a8c7-b23d3e5216fb}" />
    <EventID>30</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>30</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000001</Keywords>
    <TimeCreated SystemTime="2014-06-11T19:57:38.998656000Z" />
    <EventRecordID>5642</EventRecordID>
    <Correlation />
    <Execution ProcessID="5280" ThreadID="8472" />
    <Channel>Microsoft-Windows-CAPI2/Operational</Channel>
    <Computer>ne-r026-310cn</Computer>
    <Security UserID="S-1-5-82-1758914132-2364927631-3137608320-3227192193-3717738432" />
  </System>
  <UserData>
    <CertVerifyCertificateChainPolicy>
      <Policy type="CERT_CHAIN_POLICY_BASE" constant="1" />
      <Certificate fileRef="3E560462C61B45BE1A59F1286B34A065A878AFA0.cer" subjectName="servicebus.windows.net" />
      <CertificateChain chainRef="{19B5F58A-FA37-4213-A888-C81C340D019C}" />
      <Flags value="1000" CERT_CHAIN_POLICY_IGNORE_PEER_TRUST_FLAG="true" />
      <Status chainIndex="0" elementIndex="-1" />
      <EventAuxInfo ProcessName="w3wp.exe" />
      <CorrelationAuxInfo TaskId="{F8DE43DD-9E68-461E-8A2B-17215BA87E0C}" SeqNumber="1" />
      <Result value="800B010A">A certificate chain could not be built to a trusted root authority.</Result>
    </CertVerifyCertificateChainPolicy>
  </UserData>
</Event>

CAPI2 Build Chain event :

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-CAPI2" Guid="{5bbca4a8-b209-48dc-a8c7-b23d3e5216fb}" />
    <EventID>11</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>11</Task>
    <Opcode>2</Opcode>
    <Keywords>0x4000000000000003</Keywords>
    <TimeCreated SystemTime="2014-06-11T19:57:38.998656000Z" />
    <EventRecordID>5641</EventRecordID>
    <Correlation />
    <Execution ProcessID="5280" ThreadID="8472" />
    <Channel>Microsoft-Windows-CAPI2/Operational</Channel>
    <Computer>ne-r026-310cn</Computer>
    <Security UserID="S-1-5-82-1758914132-2364927631-3137608320-3227192193-3717738432" />
  </System>
  <UserData>
    <CertGetCertificateChain>
      <Certificate fileRef="3E560462C61B45BE1A59F1286B34A065A878AFA0.cer" subjectName="servicebus.windows.net" />
      <ValidationTime>2014-06-11T19:57:38.998Z</ValidationTime>
      <AdditionalStore />
      <ExtendedKeyUsage />
      <Flags value="0" />
      <ChainEngineInfo context="machine" />
      <AdditionalInfo>
        <NetworkConnectivityStatus value="1" _SENSAPI_NETWORK_ALIVE_LAN="true" />
      </AdditionalInfo>
      <CertificateChain chainRef="{19B5F58A-FA37-4213-A888-C81C340D019C}">
        <TrustStatus>
          <ErrorStatus value="10000" CERT_TRUST_IS_PARTIAL_CHAIN="true" />
          <InfoStatus value="0" />
        </TrustStatus>
        <ChainElement>
          <Certificate fileRef="3E560462C61B45BE1A59F1286B34A065A878AFA0.cer" subjectName="servicebus.windows.net" />
          <SignatureAlgorithm oid="1.2.840.113549.1.1.5" hashName="SHA1" publicKeyName="RSA" />
          <PublicKeyAlgorithm oid="1.2.840.113549.1.1.1" publicKeyName="RSA" publicKeyLength="2048" />
          <TrustStatus>
            <ErrorStatus value="0" />
            <InfoStatus value="2" CERT_TRUST_HAS_KEY_MATCH_ISSUER="true" />
          </TrustStatus>
          <ApplicationUsage>
            <Usage oid="1.3.6.1.5.5.7.3.2" name="Client Authentication" />
            <Usage oid="1.3.6.1.5.5.7.3.1" name="Server Authentication" />
          </ApplicationUsage>
          <IssuanceUsage />
        </ChainElement>
      </CertificateChain>
      <EventAuxInfo ProcessName="w3wp.exe" />
      <CorrelationAuxInfo TaskId="{9077AB4E-95E3-449B-AF2F-0BF42E92E6B7}" SeqNumber="11" />
      <Result value="800B010A">A certificate chain could not be built to a trusted root authority.</Result>
    </CertGetCertificateChain>
  </UserData>
</Event>

CAPI2 X509 Objects event :

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-CAPI2" Guid="{5bbca4a8-b209-48dc-a8c7-b23d3e5216fb}" />
    <EventID>90</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>90</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000200</Keywords>
    <TimeCreated SystemTime="2014-06-11T19:57:38.998656000Z" />
    <EventRecordID>5640</EventRecordID>
    <Correlation />
    <Execution ProcessID="5280" ThreadID="8472" />
    <Channel>Microsoft-Windows-CAPI2/Operational</Channel>
    <Computer>ne-r026-310cn</Computer>
    <Security UserID="S-1-5-82-1758914132-2364927631-3137608320-3227192193-3717738432" />
  </System>
  <UserData>
    <X509Objects>
      <Certificate fileRef="3E560462C61B45BE1A59F1286B34A065A878AFA0.cer" subjectName="servicebus.windows.net">
        <Subject>
          <CN>servicebus.windows.net</CN>
        </Subject>
        <SubjectKeyID computed="false" hash="BD41618C22D8DBEE9D172C12A2C549D61711ED75" />
        <SignatureAlgorithm oid="1.2.840.113549.1.1.5" hashName="SHA1" publicKeyName="RSA" />
        <PublicKeyAlgorithm oid="1.2.840.113549.1.1.1" publicKeyName="RSA" publicKeyLength="2048" />
        <Issuer>
          <CN>MSIT Machine Auth CA 2</CN>
          <DC>redmond</DC>
          <DC>corp</DC>
          <DC>microsoft</DC>
          <DC>com</DC>
        </Issuer>
        <SerialNumber>70DB015B000100008C58</SerialNumber>
        <NotBefore>2013-07-27T03:31:06Z</NotBefore>
        <NotAfter>2015-07-27T03:31:06Z</NotAfter>
        <Extensions>
          <KeyUsage value="B0" CERT_DIGITAL_SIGNATURE_KEY_USAGE="true" CERT_KEY_ENCIPHERMENT_KEY_USAGE="true" CERT_DATA_ENCIPHERMENT_KEY_USAGE="true" />
          <ExtendedKeyUsage>
            <Usage oid="1.3.6.1.5.5.7.3.2" name="Client Authentication" />
            <Usage oid="1.3.6.1.5.5.7.3.1" name="Server Authentication" />
          </ExtendedKeyUsage>
          <SubjectAltName>
            <DNSName>*.servicebus.windows.net</DNSName>
            <DNSName>servicebus.windows.net</DNSName>
          </SubjectAltName>
          <AuthorityKeyIdentifier>
            <KeyID hash="EBDB115EF8099ED8D6629CFD629DE3844A28E127" />
          </AuthorityKeyIdentifier>
        </Extensions>
      </Certificate>
      <EventAuxInfo ProcessName="w3wp.exe" />
      <CorrelationAuxInfo TaskId="{9077AB4E-95E3-449B-AF2F-0BF42E92E6B7}" SeqNumber="10" />
    </X509Objects>
  </UserData>
</Event>

Keening

The problem I also get this error very rare. But the problem I don't use console app. It's web app and both servicebus and web app in azure same resource group. – Tess

Answer 1

The missing certificates were responsible for the exception.

I haven't been able to find the certificates online and I'm still unsure of how EXACTLY they managed to install themselves BUT I think I have an idea..

How we managed to obtain the certificates? We isolated the Service Bus messaging code into a console application and executed it with admin rights on the production server. The certificates installed themselves automatically in the process.

Perhaps our application pool, running under ApplicationPoolIdentity with limited permissions was not allowing Windows to download or install the certificates.

This link seems to offer related information : http://netsekure.org/2011/04/automatic-ca-root-certificate-updates-on-windows/

Update : You can download the certificate chain here.

Keening

Thank you. In our case the identity the application pool was running under (ApplicationPoolIdentity) didn't have sufficient access levels. Assigning it an identity with elevated permissions fixed it. – Flong

@Flong nice! It really is a permission issue. Remember to change back to ApplicationPoolIdentity you don't need elevated permissions except for downloading the certificates once. – Keening

Answer 2

To eliminate certificate trust issues from Service Bus for Windows Server, use the following:

Create a list of the certificates you trust:

    var trustedCertificates = new HashSet<string>(new[]
    {
        "1245…",
        "4567…, 
        "8102…" 
    }, StringComparer.OrdinalIgnoreCase);

Trust those:

    ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, errors) =>
    {
        if (errors == SslPolicyErrors.None)
        {
            return true;
        }

        var hashString = certificate.GetCertHashString();
        var isTrusted = trustedCertificates.Contains(hashString);

        if (!isTrusted)
        {
            telemetryClient.TrackTrace($"Untrusted: {hashString} Errors: {errors} Cert: {certificate.ToString()}", SeverityLevel.Warning);
        }

        return isTrusted;
    };

Calm Service Bus down too:

    private static void SetCertificateValidator()
    {
        var retriableCertificateValidatorType = Type.GetType("Microsoft.ServiceBus.Channels.Security.RetriableCertificateValidator, Microsoft.ServiceBus", true, false);
        var instanceProperty = retriableCertificateValidatorType.GetProperty("Instance", BindingFlags.Static | BindingFlags.NonPublic);
        var instance = instanceProperty.GetValue(null);

        var peerOrChainTrustNoCheck = retriableCertificateValidatorType.GetField("peerOrChainTrustNoCheck", BindingFlags.Instance | BindingFlags.NonPublic);
        peerOrChainTrustNoCheck?.SetValue(instance, new EmptyOpX509CertificateValidator());
    }

    private sealed class EmptyOpX509CertificateValidator : X509CertificateValidator
    {
        public override void Validate(X509Certificate2 certificate)
        {
        }
    }

Vibratile