EPIPE blocks server

Question

I have written a single-threaded asynchronous server in C running on Linux: The socket is non-blocking and as for polling, I am using epoll. Benchmarks show that the server performs fine and according to Valgrind, there are no memory leaks or other problems.

The only problem is that when a write() command is interrupted (because the client closed the connection), the server will encounter an EPIPE. I am doing the interrupted artificially by running the benchmarking utility "siege" with the parameter -b. It does lots of requests in a row which all work perfectly. Now I press CTRL-C and restart the "siege". Sometimes I am lucky and the server does not manage to send the full response because the client's fd is invalid. As expected, errno is set to EPIPE. I handle this situation, execute close() on the fd and then free the memory related to the connection. Now the problem is that the server blocks and does not answer properly anymore. Here is the strace output:

epoll_wait(4, {{EPOLLIN, {u32=0, u64=0}}}, 128, -1) = 1
accept(3, {sa_family=AF_INET, sin_port=htons(55328), sin_addr=inet_addr("127.0.0.1")}, [16]) = 5
fcntl64(5, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
epoll_ctl(4, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLERR|EPOLLHUP|EPOLLET, {u32=144039912, u64=144039912}}) = 0
epoll_wait(4, {{EPOLLIN, {u32=144039912, u64=144039912}}}, 128, -1) = 1
read(5, "GET /user/register HTTP/1.1\r\nHos"..., 4096) = 161
send(5, "HTTP/1.1 200 OK\r\nContent-Type: t"..., 106, MSG_NOSIGNAL) = 106 <<<<
send(5, "00001000\r\n", 10, MSG_NOSIGNAL) = -1 EPIPE (Broken pipe)        <<<< Why did the previous send() work?
close(5)                                = 0
epoll_wait(4, {{EPOLLIN, {u32=0, u64=0}}}, 128, -1) = 1
accept(3, {sa_family=AF_INET, sin_port=htons(55329), sin_addr=inet_addr("127.0.0.1")}, [16]) = 5
...

(I deleted the printf()'s from the log in case you're wondering)

As you can see, the client establishes a new connection which consequently is accepted. Then, it's added to the EPOLL queue. epoll_wait() signalises that the client sent data (EPOLLIN). The request is parsed and and a response is composed. Sending the headers works fine but when it comes to the body, write() results in an EPIPE. It is not a bug in "siege" because it blocks any incoming connections, no matter from which client.

#include "Connection.h"

static ExceptionManager *exc;

void Connection0(ExceptionManager *e) {
    exc = e;
}

void Connection_Init(Connection *this) {
    Socket_Init(&this->server);
    Socket_SetReusableFlag(&this->server);
    Socket_SetCloexecFlag(&this->server, true);
    Socket_SetBlockingFlag(&this->server, true);
    Socket_ListenTCP(&this->server, 8080, SOMAXCONN);

    // Add the server socket to epoll
    Poll_Init(&this->poll, SOMAXCONN, (void *) &Connection_OnEvent, this);
    Poll_AddEvent(&this->poll, NULL, this->server.fd, EPOLLIN | EPOLLET | EPOLLHUP | EPOLLERR);

    this->activeConn = 0;
}

void Connection_Destroy(Connection *this) {
    Poll_Destroy(&this->poll);
    Socket_Destroy(&this->server);
}

void Connection_Process(Connection *this) {
    Poll_Process(&this->poll, -1);
}

void Connection_AcceptClient(Connection *this) {
    Client *client;

    SocketConnection conn = Socket_Accept(&this->server);
    SocketConnection_SetBlockingFlag(&conn, true);

    client = New(Client);

    client->req = NULL;

    client->conn = New(SocketConnection);
    client->conn->remote = conn.remote;
    client->conn->fd = conn.fd;

    this->activeConn++;

    Poll_AddEvent(&this->poll, client, conn.fd, EPOLLIN | EPOLLET | EPOLLHUP | EPOLLERR);
}

void Connection_DestroyClient(Connection *this, Client *client) {
    // Poll_DeleteEvent(&this->poll, client->conn->fd);
    SocketConnection_Close(client->conn);

    if (client->req != NULL) {
        Request_Destroy(client->req);
        Memory_Free(client->req);
    }

    if (client->conn != NULL) {
        Memory_Free(client->conn);
    }

    Memory_Free(client);

    this->activeConn--;
}

void Connection_OnEvent(Connection *this, int events, Client *client) {
    /* error or connection hung up */
    if (client != NULL && (events & (EPOLLHUP | EPOLLERR))) {
        String_Print(String("EPOLLHUP | EPOLLERR received\n"));
        Connection_DestroyClient(this, client);
        return;
    }

    /* incoming connection */
    if (client == NULL && (events & EPOLLIN)) {
        if (this->activeConn > SOMAXCONN - 1) { /* TODO */
            String_Print(String("Too many connections...\n"));
            return;
        }

        Connection_AcceptClient(this);
        return;
    }

    /* receiving data from client */
    if (client != NULL && (events & EPOLLIN)) {
        if (client->req == NULL) {
            client->req = New(Request);
            Request_Init(client->req, client->conn);
        }

        bool keepOpen = false;

        try (exc) {
            keepOpen = Request_Parse(client->req);
        } catch(&SocketConnection_PipeException, e) {
            printf("Caught PipeException on fd=%d\n", client->conn->fd); fflush(stdout);
        } catch(&SocketConnection_ConnectionResetException, e) {
            printf("Caught ConnectionResetException on fd=%d\n", client->conn->fd); fflush(stdout);
        } finally {
            if (!keepOpen) {
                printf("Will close...\n"); fflush(stdout);
                Connection_DestroyClient(this, client);
            }
        } tryEnd;
    }
}

Laurent

Ok, I added the crucial code extract. Unfortunately, the rest is too much to post it here. – Laurent

What do you mean by "new clients are blocked"? Can you connect to the server again? – Jilolo

Yes, I can connect but every send() results in a EPIPE and none of the clients get an answer. – Laurent

Can you try reproducing the issue with "exception handling" removed? Is that btw setjmp-based? – Jilolo

If you use epoll in edge triggered mode, you have to read until EWOULDBLOCK occurs, or you might not receive events unless you happen to empty the entire buffer. I'd investigate how edge triggering plays allong with accept() as well. – Lakin

The bug also happens without exception handling and yes, it's setjmp-based. – Laurent

nos, with all EPOLLET's removed, the code runs perfectly. Thanks. :) – Laurent

It suffices to remove only the first EPOLLET which is responsible for adding the server's file descriptor to the epoll queue. The clients' file descriptors can stay edge-triggered. – Laurent

Answer 1

Use sigaction() to set the action for SIGPIPE to SIG_IGN. Then you will just get the return code of -1 with errno set to EPIPE, without the signal.

On Linux, an alternative is to use send() with the MSG_NOSIGNAL flag, instead of write(). This allows you to suppress the signal for that write, without affecting any others. An alternative on BSD systems is the SO_NOSIGPIPE socket option, which suppresses SIGPIPE for all writes on that socket.

At any time the kernel TCP implementation may receive a TCP RST from the peer. The next write on the socket after that point will result in the EPIPE error, and the SIGPIPE signal if it was not suppressed. So even with two consecutive writes, the first could be successful and the next one could fail with EPIPE and SIGPIPE.

Update: Although it is not part of the code that was posted, I see in your strace output that you are calling fcntl64(5, F_SETFL, O_RDONLY|O_NONBLOCK). Since O_RDONLY is 0, probably your code just sets the flags to O_NONBLOCK. It should get the current flags and then set it to OldFlags | O_NONBLOCK in order to set non-blocking mode. Setting a socket to read-only mode seems likely to cause a problem when writing. Or it may be that you accidentally used F_GETFD instead of F_GETFL to get the old flags.

Exhale

SIGPIPE is already set to SIG_IGN. Otherwise the application would stop running if it encountered a SIGPIPE which is not the case. The problem must be somewhere else. Anyway, thanks for the suggestion of using send() with MSG_NOSIGNAL. That's indeed better than ignoring SIGPIPE globally. – Laurent

Unfortunately, moving to send() did not fix the issue. New clients are still "blocked": send(5, "00001000\r\n", 10, MSG_NOSIGNAL) = -1 EPIPE (Broken pipe) – Laurent

By the way, SocketConnection_PipeException is an exception that is raised in SocketConnection_Write() when errno returns EPIPE after the send(). I tried to ignore EPIPE errors by not throwing that exception but still no change. – Laurent

I'm not sure what you mean when you say that new clients are "blocked". Are you not calling epoll_wait() again after the EPIPE? You did not post the code that calls this and your strace stops at the EPIPE. – Exhale

Also I do not see where you are clearing the old client from the epoll descriptor, or perhaps you just did not post that code? At least include more strace output, showing the closing of the old socket, the epoll_ctl to remove the old socket, and the epoll_wait to wait for the next event. The epoll_wait is important to include since that is where the new client connection should arrive. – Exhale

Sure, epoll_wait() is called after EPIPE. Connection_Process() is in a infinite loop and Poll_Process() calls epoll_wait() accordingly. I just deleted it from the strace log. – Laurent

The old client does not need to be removed from the epoll descriptor as the kernel does this automatically upon close(). I had the Poll_DeleteEvent() line uncommented, which calls the epoll_ctl() for removing the fd, but it did not have any effect. I updated the log as requested. – Laurent

Closing a file descriptor removes it from epoll set automatically. – Jilolo

I was also wondering about that O_RDONLY in the log. You were right, I accidentally used F_GETFD instead of F_GETFL to get the old flags. I've updated the log. – Laurent

Answer 2

I can't tell from your code (possibly because I've not looked in the right place) whether you adjust the epoll structure after you get the SIGPIPE from one of the descriptors. After you get a SIGPIPE (or EPIPE) from a file descriptor, you will get a repeat on any subsequent use of the same file descriptor. You'd need to close the file descriptor, and then adjust your server's internal structures appropriately.

Clova

You mean I've to remove the file descriptor off the epoll queue? Actually this is not necessary, because it suffices to close the file descriptor which is done in SocketConnection_Close(). – Laurent

If you're sure that closing it is all that's necessary, that's fine. But there's an epoll_ctl() call to remove a descriptor from the epoll list. However, my 'answer' is not much more than semi-educated guesswork. – Clova