ECONNRESET in Send Linux C

Question

According to Unix Network Programming when a socket writes twice to a closed socket (after a FIN packet), then in the first time it succeeded to send, but receives an RST packet from the other host. Since the host receives an RST, the socket is destroyed. Thus in the second time it writes, the SIGPIPE signal is received, and an EPIPE error is returned.

However, in send man pages ECONNRESET can be returned, which means that an RST packet is received. When it returns ECONNRESET -there no signal is returned.

What are the cases ECONNRESET can be returned? and why does there is no SIGPIPE signal in this case?

Note: I have checked I similar question here. However, when I run in my linux computer, send returned the EPIPE error, and not ECONNRESET.

Votyak

If you set SIGPIPE to be ignored you get the error instead of the signal. It's all in the book. – Darondarooge

By the book I cited , and by in my Linux computer, I got EPIPE, not ECONNRESET . – Votyak

Answer 1

If the peer closed the connection while there were still unhandled data in the socket buffer it will send a RST packet back. This will cause a flag to be set on the socket and the next send will return ECONNRESET as the result . EPIPE instead is returned (or SIGPIPE triggered) on send if the connection was closed by the peer with no outstanding data. In both cases the local socket is still open (i.e. the file descriptor is valid), but the underlying connection is closed.

Example: Imagine a server which reads a single byte and then closes the connection:

• EPIPE: The client sends first one byte. After the server read the byte and closed the connection the client will send some more data and then again some data. The latest send call will trigger EPIPE/SIGPIPE.
• ECONNRESET: The client sends first more than one byte. The server will read a single byte and close the connection with more bytes in the sockets receive buffer. This will trigger a connection RST packet from the server and on the next send the client will receive ECONNRESET.

Cammycamomile

Thanks :) I confirm your comment in my linux. How does the client distinguish between the two cases? I think that in the second case an RST ACK packet is sent, but I am not sure. Why ECONNRESET does not have a signal? – Votyak

@user3563894: like I have written, the RST packet from the peer causes the ECONNRESET. And why one situation triggers a signal and the other not: this decision was made probably about 40 years ago. – Cammycamomile

But in both cases (EPIPE and ECONNRESET) an RST packet is returned. How can the client know if the server read a single byte or more? – Votyak

@user3563894: the client does not have the knowledge. The difference is that in the case of ECONNRESET only the RST is sent by the peer (hard close with data inside socket buffer) while in case of EPIPE first a FIN is sent (normal close). The RST is only sent after the peer received more data. And this difference (RST vs. FIN followed by RST) can be seen at the client side. – Cammycamomile

Thanks :) Do you know a good reference where I can read on these edge cases? – Votyak

@user3563894: UNIX Network programming Volume 1 is an extensive resource which covers also these things. – Cammycamomile

Answer 2

A TCP connection can be seen as two data pipelines between two endpoints. One data pipeline for sending data from A to B and one data pipeline for sending data from B to A. These two pipelines belong to a single connection but they don't otherwise influence each other. Sending data on one pipeline has no effect on data being sent on the other pipeline. If data on one pipeline is reply data to data sent previously on the other pipeline, this is something only your application will know, TCP knows nothing about that. The task of TCP is to make sure that data reliably makes it from one end of the pipeline to the other end and that as fast as possible, that is all that TCP cares for.

As soon as one side is done sending data, it tells the other side it is done by tranmitting it a packet with the FIN flag set. Sending a FIN flag means "I have sent all the data I wanted to send to you, so my send pipeline is now closed". You can trigger that intentionally in your code by calling shutdown(socketfd, SHUT_WR). If the other side will then call recv() on the socket, it won't get an error but receive will say that it read zero bytes, which means "end of stream". End of stream is not an error, it only means that no more data will ever arrive there, no matter how often you are going to call recv() on that socket.

Of course, this doesn't affect the other pipeline, so when A -> B is closed, B -> A can still be used. You can still receive from that socket, even though you closed your sending pipeline. At some point, though, also B will be done with sending data and also transmit a FIN. Once both pipelines are closed, the connection as a whole is closed and this would be a graceful shutdown, as both sides have been able to send all the data they wanted to send and no data should have been lost, since as long as there was unconfirmed data in flight, the other side would not have said it is done but wait for that data to be reliably transferred first.

Alternatively there is the RST flag which closes the entire connection at once, regardless if the other side was done sending and regardless if there was unconfirmed data in flight, so a RST has a high potential of causing data to be lost. As that is an exceptional situation that may require special handling, it would be useful for programmers to know if that was the case, that's why there exists two errors:

EPIPE - You cannot send over that pipe as that pipe is not valid anymore. However, all data that you were sending before it broke was still reliably delivered, you just cannot send any new data.

ECONNRESET - Your pipe is broken and it may be the case that data you were trying to send before got lost in the middle of transfer. If that is a problem, you better handle it somehow.

But these two errors do not map one to one to the FIN and RST flag. If you receive a RST in a situation where the system sees no risk of data loss, there is no reason to drive you round the bend for nothing. So if all data you sent before was ACKed to be correctly received and then the connection was closed by a RST when you tried to send new data, no data was lost. This includes the current data you tried to send as this data wasn't lost, it was never sent on the way, that's a difference as you still have it around whereas data you were sending before may not be around anymore. If your car breaks down in the middle of a road trip then this is quite a different situation than if you you are still at home as your car engine refused to even start. So in the end it's your system that decides if a RST triggers a ECONNRESET or a EPIPE.

Okay, but why would the other side send you a RST in the first place? Why not always closing with FIN? Well, there exists a couple of reasons but the two most prominent ones are:

1. A side can only signal the other one that it is done sending but the only way to signal that it is done with the entire connection is to send a RST. So if one side wants to close a connection and it wants to close it gracefully, it will first send a FIN to signal that it won't send new data anymore and then give the other side some time to stop sending data, allowing in-flight data to pass through and to finally send a FIN as well. However, what if the other side doesn't want to stop and keeps sending and sending? This behavior is legal as a FIN doesn't mean that the connection needs to close, it only means one side is done. The result is that the FIN is followed by RST to finally close that connection. This may have caused in-flight data to be lost or it may not, only the recipient of the RST will know for sure as if data was lost, it must have been on his side since the sender of the RST was surely not sending any more data after the FIN. For a recv() call, this RST has no effect as there was a FIN before signaling "end of stream", so recv() will report having read zero bytes.

2. One side shall close the connection, yet it sill has unsent data. Ideally it would wait till all unsent data has been sent and then transmit a FIN, however, the time it is allowed to wait is limited and after that time has passed, there is still unsent data left. In that case it cannot send a FIN as that FIN would be a lie. It would tell the other side "Hey, I sent all the data I wanted to send" but that's not true. There was data that should have been sent but as the close was required to be instant, this data had to be discarded and as a result, this side will directly send a RST. Whether this RST triggers a ECONNRESET for the send() call depends again on the fact, if the recipient of the RST had unsent data in flight or not. However, it will for sure trigger a ECONNRESET error on the next recv() call to tell the program "The other side actually wanted to send more data to you but it couldn't and thus some of that data was lost", since this may again be a situation that handling somehow, as the data you've received was for sure incomplete and this is something you should be made aware of.

If you want to force a socket to be always closed directly with RST and never with FIN/FIN or FIN/RST, you can just set the Linger time to zero.

struct linger l = { .l_onoff = 1, .l_linger = 0 };
setsockopt(socketfd, SOL_SOCKET, SO_LINGER, &l, sizeof(l));

Now the socket must close instantly and without any delay, no matter how little and the only way to close a TCP socket instantly is to send a RST. Some people think "Why enabling it and setting time to zero? Why not just disabling it instead?" but disabling has a different meaning.

The linger time is the time a close() call may block to perform pending send actions to close a socket gracefully. If enabled (.l_onoff != 0), a call to close() may block for up to .l_linger seconds. If you set time to zero, it may not block at all and thus terminates instantly (RST). However, if you disable it, then close() will never block either but then the system may still linger on close, yet this lingering happens in the background, so your process won't notice it any longer and thus also cannot know when the socket has really closed, as the socketfd becomes invalid at once, even if the underlying socket in kernel still exists.

Houlihan

Here's what I'm confused about, and can't find any definitive reference anywhere: if receiving side has accumulated pending read data (packets arrived, but recv() hasn't picked up the data), and say there's 4KB pending in receive data, and then RST arrives. When application then calls recv(1KB), will it get the pending bytes from the buffer, or will it receive an error because RST superseeds pending data and essentially received data is discarded and lost (even though it did make it to receiver's TCP stack)? – Frederic

@Frederic Keep in mind that there are protocol standards like TCP described in RFCs, there are implementations of these standards in system kernels, and there are user space APIs (like POSIX) to make use of these kernel implementations; and the first ones don't dictate how the last ones are supposed to operate, so different APIs can exist for the same protocol and show different behavior. And POSIX API is generic, it doesn't dictate exactly how to behave for certain TCP flags, that's up to the operating system implementation. So it's hard to give a definitive answer for all systems. – Houlihan

@Frederic There is even an RFC (2525) which collected a list of known problems caused by certain TCP implementations. There is nothing in any standard that would force an implementer to drop already received data if a RST is received. On the other hand, it's not forbidden to do that either but I would not expect it under normal conditions. Yet if an implementation for some reason cannot remember the state of having received a RST unless the socket is destroyed, the data will be lost as part of destroying the socket. – Houlihan

Interesting. The context for question is this thread that I started, thinking that it should be possible to pick up queued response data, but now I have doubts. How do you interpret RFC 793, page 69 -- as drop or deliver pending? – Frederic

@Frederic RFC 793 only talks about removing all data from the "retransmission queue", which is the content in the send buffer (once a RST is received, no data should be sent to the peer anymore). It also says delete the TCB but it's not defined if the receive buffer is part of the TCB or not; personally I would say it isn't. Once received data has been acknowledged, TCP considers it as "successfully received", regardless if your app code has ever seen it or not. – Houlihan

@Frederic Think about it that way: Before data is placed into the receive buffer, it is acknowledged to the other side and that means "Yes, I have successfully received it". When the other side now sends a RST, it doesn't expect that "already received" data will vanish but that's exactly the case if the receive buffer was dropped, as then your app had no chance to see data that was told to be correctly received to the other side. – Houlihan

Yes, that makes sense to me, which is why in the above context thread, I thought received response should still be readable. But examples there (I haven't tested them yet) seem to suggest otherwise. – Frederic

@Frederic Okay, I commented on that GitHub thread and wrote a small C program that proves that already sent and acknowledged content is not getting lost, at least no on FreeBSD, Linux, or macOS (I currently have no Windows test machine available where I could test that). Only still unsent data may be lost as that is an actual requirement by the RFC. And yes, the RFC doesn't explicitly forbid received data to be lost but as I pointed out before, this seems illogical and goes against the idea behind TCP as being "reliable". – Houlihan