What causes "Tcp channel protocol violation expecting preamble" in .NET Remoting?

Question

I have a distributed application that uses .NET Remoting on an in-house gigabit network. There is a single server, and over a dozen clients that connect to the server. The clients run multiple threads, and there can be up to 10 concurrent requests from each client.

This application works very well most of the time. The server stays up for months at a time. From time to time I get an exception on a client, and I can't figure out what's causing the exception. The exception and stack trace are:

System.Runtime.Remoting.RemotingException: Tcp channel protocol violation: expecting preamble.

Server stack trace: 
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

That's the stack trace back to where I make the remoting call.

I've looked and can find nothing out of the ordinary in the calls I'm making, or in the data that the server is returning.

Google searches for this error are not very fruitful. Most of the errors I've seen revolve around somebody converting from HTTP to TCP and not changing everything, so they get the exception when they try to connect. In my case, the client will run for days before I get this error.

One other data point: the server does get a lot of requests. Most of the clients are Web crawlers that each make more than 2,000 requests to the server every minute. So the server is processing upwards of 500 requests per second, with bursts of higher traffic. In any case, the server seems to handle the traffic okay, and I would expect a much different error if the server became overloaded.

Any ideas what's causing this error?

Huda

Answer 1

This error usually happens when a message received with wrong headers. You can repeat this error with creating a telnet connection to your server and type something. In most cases it's a network error.

I highly recommend the check your firewall. Some firewalls drops the network packets due wrong attack alert.

Load balancing is another possible reason. Load balancer splits packets to different servers.

Vargas

It could very well be the firewall. I push a lot of traffic through it and although it can easily handle the average traffic volume, perhaps there are bursts that overrun its buffers or something. I'm working on some modifications to my code that will smooth out the traffic in order to eliminate the worst of those bursts. – Huda

If any property exist on a marshallbyref object, you can try to convert them to methods. Each time you access the property it's causes a message call. Converting them to methods reduces the messages between client and server. Also you may serialize some classes for client caching. – Vargas

There aren't any accessible properties on the remote object. It looks like it's the switch. I'm pushing an incredible number of packets through this switch, and it's looking like the switch just can't handle it. In any case, I'm convinced that the error isn't a software failure, but rather a hardware issue. Thanks for the response. – Huda

Answer 2

I've see this issue when MS DNS server used too lot of open port, so outbound port can't be created. Yes, it's very funny.

Chessman