EPOLLRDHUP not reliable

Question

I'm using non-blocking read/writes over a client-server TCP connection with epoll_wait.

Problem is, I can't reliably detect 'peer closed connection' event using the EPOLLRDHUP flag. It often happens that the flag is not set. The client uses close() and the server, most of the time, receives, from epoll_wait, an EPOLLIN | EPOLLRDHUP event. Reading yields zero bytes, as expected. Sometimes, though, only EPOLLIN comes, yielding zero bytes.

Investigation using tcpdump shows that normal shutdown occurs as far as I can tell. I see a Flags [F.], Flags [F.], Flags [.] sequence of events, which should correspond to FIN, FIN and ACK. SO_LINGER is nowhere used.

I considered handling 'peer closed' on zero-byte read, however, there is the possibility that you get an EPOLLIN | EPOLLRDHUP event with non-zero bytes available, when the peer sends & immediately closes the connection - case in which I need to base myself on the EPOLLRDHUP. Suggestions?

George

Answer 1

To answer this: EPOLLRDHUP indeed comes if you continue to poll after receiving a zero-byte read. So from my experiments it looks like either an EPOLLIN with zero-byte read or an EPOLLRDHUP are reliable indicators for orderly shutdown, the only trouble was, they are not received together. Sometimes (the case that makes the subject of this question), it happens that EPOLLIN is received, yielding zero bytes (connection terminated), and on subsequent polling you get to see the EPOLLRDHUP. Other times, it's vice-versa: you get the EPOLLRDHUP together with an EPOLLIN that signals actual bytes to be read. Then, on subsequent reads, you get zero bytes.

George

So the upshot is that if the EPOLLIN | EPOLLRDHUP combination is received, which signals that that there are non-zero bytes to read and the connection was terminated, just keep reading until you get 0 bytes -- and that will signal shutdown. In other words, when you read 0, you're done -- so it suffices just to check for EPOLLIN? – Thirtieth

If you get EPOLLIN | EPOLLRDHUP and then you read() or recv() into a buffer of sized X, but get back less than X bytes, then EPOLLRDHUP can be leveraged to indicate EOF without an additional read. If you get back X bytes, then read again until you get less than X bytes. Even if the connection has closed on the remote side, there might still be data pending in local kernel socket buffers that you might not have read yet. – Avera