How do I troubleshoot PipeExceptions and CommunicationExceptions in WCF?

Question

I have an application consisting of several WCF services, some of which are implemented in Workflow Foundation (.NET 3.5), others just plain C#. These services communicate with each other over a netNamedPipeBinding for performance reasons. The trouble is that I'm seeing more and more CommunicationExceptions and underlying PipeExceptions as soon as the system load increases. The funny thing is that these transactions do seem to finish ultimately. One reason for that is that we have a retry mechanism in the workflows, but even the calls from plain C# services succeed even though I see these errors in the WCF traces. Is there some retry mechanism in the named pipe subsystem of Windows or something?

However I want these errors fixed, or at least understand the underlying problem. I feel they are impacting the performance and stability of the application. How do I go about and diagnose the root cause of these errors correctly if I don't see any other exception coming from the services themselves?

Here are some of the exceptions I get:

PipeException: There was an error reading from the pipe: The pipe has been ended. (109, 0x6d).

And:

PipeException: The operation cannot be completed because the pipe was closed. This may have been caused by the application on the other end of the pipe exiting.

within a TimeoutException: The pipe connection was aborted because an asynchronous read from the pipe did not complete within the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout.

Now the timeout exception seems to come from the fact that the system is having trouble handling the load. These operations are quite small normally but the amount of them seems to be the problem. Or could this the result of previous pipe connections being terminated and not given back to the pool?

I've tried experimenting with a serviceThrottling behavior in WCF config to increase the amount of instances etc. but these errors keep emerging. Any tips?

/EDIT: I did turn on both WCF tracing and message logging. That's where I'm seeing the PipeExceptions and CommunicationExceptions. The application itself is not showing any errors. We've instrumented the WCF services quite a bit so that all exceptions get logged using log4net and I don't see any errors in those logs at all. This all seems to be happening at WCF level.

Ultranationalism

We ultimately found out that there are problems in the WCF net.pipe stack. We changed to a basicHttp binding and besides having all errors disappear, the service calls became at least twice as fast. Really weird. – Ultranationalism

how did you ultimately find this out? I'm running into similar problems but I'm not quite ready to jump ship yet to basicHttp. – Visage

@ColeW whoo, this is an old question, I have to dig back into my memory. I think we got some feedback on the Regional Directors mailing list, my colleague is an RD that the net.pipe stack wasn't stable under high loads and after testing with basicHttp, we got much better throughput and more stable services. I've long since abandoned WCF so I'm not sure about the current state. Sad to hear you're still having the same problems. – Ultranationalism

yes it is a very old question! We actually abandoned wcf net.tcp bindings for similar reasons so this is our second binding for communication between 2 applications on the same machine.Wcf seems as though it's not all it's cracked up to be. We may have to end up switching to basicHttp. It's weird though that all my tests on my local machine turn out fine. I can hit the service 10x as hard and 10x as many times and I get no errors. – Visage

Answer 1

You may want to activate logging and activity tracing on your service. Then use the Service Trace Viewer to have a clear view of the sequence of events.

This will give a huge amount of data to go through so I suggest you activate it just for one test case, and then deactivate it. This should limit the volume of data.

Scission

Thanks, indeed I did that already. That's actually where I am seeing the PipeExceptions and not necessarily in my application. – Ultranationalism