I got the failed communication error while trying to set up DTC and MSMQ on a cluster. In my case the underlying error was "Ran out of memory." I was able to send transactional messages from the cluster to another server, but not back from that server to the cluster. My service would throw this exception:
System.Transactions.TransactionAbortedException: The transaction has aborted.
---> System.Transactions.TransactionManagerCommunicationException: Communication
with the underlying transaction manager has failed. --->
System.Runtime.InteropServices.COMException: Ran out of memory (Exception from HRESULT: 0x80000002)
This article had the very obscure solution: http://www.nervousadmin.com/category/microsoft/windows/dtc/
To summarize:
There is a guid in the registry for the key ClusterDefaultResource under HKLM\Cluster\ResourceTypes\Distributed Transaction Coordinator that needs to align with the guid argument on the DTC service's path to executable.
Another symptom of this issue is that you would get an out of memory error if you try to access the DTC properties via the Component Services management console. Look in the console tree under Component Services/Computers/My Computer/Distributed Transaction Coordinator and right click on each of the DTCs listed there. This will throw the error if your guids are not aligned.
- Open services.msc. Find the Distributed Transaction Coordinator (if there are two, you're looking for the one with the guid in its name)
- Open that DTC's properties. Copy the guid from the 'path to executable'
- Open regedit. Find HKLM\Cluster\ResourceTypes\Distributed Transaction Coordinator
- Compare the ClusterDefaultResource value to the guid you've copied. If they are different, the next step should fix things. If not, this isn't your answer.
- Back up the current value. Edit the ClusterDefaultResource property: paste the guid you copied from the services.msc DTC properties. You will need to do this on each node in your cluster.
- With luck, this has solved your problem.