Intermittent MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system [servername]

Question

We are occasionally seeing the error "MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system [servername]" on our servers. We are using NServiceBus.

After we reboot the server, we are ok, things work perfectly. However, after an undetermined amount of time, this error randomly happens and the services stop working and everything comes to a halt.

Has anyone encountered this before?

Spermatozoid

What do your security settings look like on DTC for those machines? Are they in the same domain? – Gigigigli

Anything in the system logs around that time? – Barrow

There is nothing abnormal in the security logs at the time. What's interesting is that the NServiceBusHost.exe dies first, and then 30 milliseconds later, I get the MSDTC error. – Spermatozoid

Here are our DTC security settings: [x] Network DTC Access [x] Allow Remote Clients [x] Allow Remote Administration [x] Allow Inbound [x] Allow Outbound [x] Mutual Authentication Required [x] Enable SNA LU 6.2 Transactions DTC Logon Account Account: NT AUTHORITY\NetworkService Yes, these are all on the same domain. – Spermatozoid

You might have checked this already, but have you confirmed that the MSDTC service did not shut down or restart on either the machine running NServiceBus or on [servername]? You mentioned security logs, but did you also check the Windows Event Log? Also, do you have a firewall between the two(?) machines or have antivirus running? – Bootjack

@CraigGambino did you figure out the cause of this issue? I am experiencing the exact same scenario you mentioned – Smidgen