Random "An existing connection was forcibly closed by the remote host." after a TCP reset

Question

I have two parts, a client and a server. And I try to send data (size > 5840 Bytes) from the client to the server and then the server sends the data back. I loop this a number of times waiting a second between each time. Sometime the server application crash, the crash seems very random the error:

Unhandled Exception: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. --->

System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)

at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 s ize)

--- End of inner exception stack trace ---

at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 s ize)

at TCP_Server.Program.Main(String[] args)

Client code (This is inside the loop):

            try
            {
                Int32 port = 13777;
                using (TcpClient client = new TcpClient(ip, port))
                using (NetworkStream stream = client.GetStream())
                {
                    client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
                    var data = GenerateData(size);

                    sw.Start();
                    // Send the message to the connected TcpServer. 
                    stream.Write(data, 0, data.Length);

                    // Buffer to store the response bytes.
                    data = new Byte[size];

                    // Read the first batch of the TcpServer response bytes.
                    Int32 bytes = stream.Read(data, 0, data.Length);

                    sw.Stop();
                    Console.WriteLine(i + ": Done transporting " + size + " bytes to and from " + ip + " time: " +
                                      sw.ElapsedMilliseconds + " ms");
                    // Close everything.
                    stream.Close();
                    client.Close();
                }
                
            }
            catch (ArgumentNullException e)
            {
                Console.WriteLine("ArgumentNullException: {0}", e);
            }
            catch (SocketException e)
            {
                Console.WriteLine("SocketException: {0}", e);
            }

            sw.Reset();
        

Server code:

            Byte[] bytes = new Byte[size];

            // Enter the listening loop. 
            for (int i = 0; i < numberOfPackages; i++)
            {
                using (TcpClient client = server.AcceptTcpClient())
                using (NetworkStream stream = client.GetStream())
                {
                    client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
                    // Loop to receive all the data sent by the client. 
                    while ((stream.Read(bytes, 0, bytes.Length)) != 0)
                    {
                        // Send back a response.
                        stream.Write(bytes, 0, size);

                    }
                    client.GetStream().Close();
                    client.Close();
                }
                

                Console.WriteLine("Receive data size " + size);

            }

I have used wireshark to monitor the tcp packages sent and found that a TCP RST is sent from the client to the server before the program crashes. So I assume the problem is that the the RST is not handles correctly. There is no firewall between the client and the host so that is not the problem.

The wireshark files for both the Client and Server is here: https://www.dropbox.com/sh/ctl2chq3y2c20n7/AACgIJ8IRiclqnyOyw8sqd9La?dl=0

So either I need to get rid of the TCP RST or I need my server to handle it in some way and not crash.

I have tried to use longer waiting time but it does not help. If the data is below 5840 Bytes I do not get any crashes, what I know of.

Any suggestions or ideas?

Theoretician

Answer 1

Two problems that I see:

1. You are assuming that a read for size bytes will actually read size bytes. It does not. TCP is a streaming protocol. A read reads at least one byte. That's the only guarantee given.
2. Your code will randomly deadlock. The client writes data to the server. The server echos it back. But the client will not read until it has written everything. If it writes more than the network buffers can take this will deadlock. The client must read and write concurrently. Probably, you need another thread/task for reading the data back. A good pattern to solve that issue would be to start one writer task, one reader task and to Task.WhenAll/WaitAll to join them back.

I'm not sure under what exact circumstances a TCP stack would send a RST. Could be due to the deadlock causing a timeout.

You are not swallowing any exceptions, right?

Normally, closing a connection performs an orderly shutdown in the background. But I'm unsure about what happens when the other side is still writing at this point. Maybe the answer is that an RST is generated by the receiver of the writes. Surely, the TCP spec would answer this question. If this answer is to be trusted then indeed a Shutdown(Read)/Close followed by an incoming write would RST the connection.

Fix both issues and report back with your findings.

Pivotal

Thank you!!! Since I only need my application to send and then receive and not do it at the same time, I only needed to fix the first problem. – Theoretician

@Theoretician I think you did not understand the deadlock problem. This code will randomly deadlock. Try sending 100MB and this will deadlock for sure. – Pivotal

Even if I first send everything from the client, then the server receives everything before sending it back? After this i do a wait and do everything again. The code is not used for real it is more of a benchmark application. – Theoretician

The server echoes the data back but the buffer fills up because nobody drains it. network buffers are limited in size. Run 100MB through this. – Pivotal

I'm only able to run 10000000 Byte =9.536743 (MB) because of other memory problems. And that work fine. – Theoretician

You can't test for this reliably because the deadlock is non-deterministic. You still don't understand why the deadlock exists. Doesn't it bother you that there might be non-deterministic bug in you app?! Anyway, if reliability is not important because this is a benchmark tool then it's probably not worth investigating this further. – Pivotal

But will not my change by moving the write of the server after the receive is done, remove this? – Theoretician

Yes, did you make that change? I did not see that. That fixes it. – Pivotal

Answer 2

We had this problem when using HttpClient random errors like

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
 ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

or

 ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.

Problem in Explicit Congestion Notification, or ECN, in the TCP stack. This is also known as ECN Capability. Explicit Congestion Notification is an extension to the Internet Protocol and to the Transmission Control Protocol and is defined in RFC 3168. ECN allows end-to-end notification of network congestion without dropping packets. If ECN Capability is enabled, you can disable it:

netsh interface tcp set global ecncapability=disabled

A reboot is not required.

Chantress

Answer 3

_{Solution by OP migrated from the question to an answer:}

Thanks to the answers I got it to work with the following changes:

Server:

// Loop to receive all the data sent by the client.
int k = 0;
while (k < size)
{
   int bytesRead = stream.Read(bytes, 0, bytes.Length);
   k += bytesRead;
}
// Send back a response.
stream.Write(bytes, 0, size);

And the same when receiving on the client side. Since I first want to send all data and then let the server respond this works for my application.

Doralia

Answer 4

I am no expert but don't you think calling stream.Close() on Client will close stream while server is still trying to write at

// Send back a response.
stream.Write(bytes, 0, size);

Also you may want to put some data to mark end of it, so that server knows and stops reading.

Shel

Might be, do you have any idea what to change? I have been stuck with this problem for 2 days now so I'm really tired of it. – Theoretician

Either send some header which may contain length of data or append some unique sequence to data to mark end of it. In server process this to know when server is supposed to end reading. Also implement suggestions by usr in other answer. Or just work on only one way transfer first, so that you can pin point the problem. – Shel

And to avoid complications, you can read and write on different ports, if that's doable for you. – Shel

On how to mark end you can read here: How should I mark the end of a TCP packet? – Shel

Answer 5

I solved the same problem with increasing cache limit of IIS root entry. you can also disable caching.

Mare