Is level triggered or edge triggered more performant?

Question

I am trying to figure out what is more performant, edge triggered or level triggered epoll.

Mainly I am considering "performant" as:

1. Ability to handle multiple connections without degredation.

2. Ability to keep the uptmost speed per inbound message.

I am actually more concerned about #2, but #1 is also important.

I've been running tests with a single threaded consumer (accept/read multiple socket connections using epoll_wait), and multiple producers.

So far I've seen no difference, even up to 1000 file descriptors.

I've been laboring under the idea (delusion?) that edge triggered should be more performant because less interupts will be received. Is this a correct assumption?

One issue with my test, that might be masking performance differences, is that I don't dispatch my messages to threads once they are received, so the less interrupts don't really matter. I've been loath to do this test because I've been using __asm__ rdtsc to get my "timestamps," so I don't want to have to reconcile what core my original timestamp came from.

What makes me even more suspicious is that level triggered epoll performs better on some benchmarks I've seen.

Which is better? Under what circumstances? Is there no difference? Any insights would be appreciated.

My sockets are non-blocking.

Doro

Hard to tell.. depends on the application. You have to measure it, and pick the one which suits you better. – Larry

The main difference is that edge triggered works well with non-blocking sockets while level triggered works well with blocking sockets. – Mountainous

I am using non-blocking, for reference. – Doro

Answer 1

I wouldn't expect to see a huge performance difference between edge and level triggered.

For edge-triggered you always have to drain the input buffer, so you have one useless (just returning EWOULDBLOCK) recv syscall. But for level triggered you might use more epoll_wait syscalls. As the man page points out, avoiding starvation might be slightly easier in level triggered mode.

The real difference is that when you want to use multiple threads you'll have to use edge-triggered mode (although you'll still have to be careful with getting synchronization right).

Voletta

For multiple threads oneshot semantics (EPOLLONESHOT) might be useful as well. – Retha

EPOLLONESHOT can simplify synchronization in a multithreaded environment, but it is not necessary. – Voletta

Can you not use multiple threads using level triggered? What's the advantage here? – Doro

The main problem with level triggered mode and multiple threads is that a single fd being ready would wake up all waiting threads. Now, thinking about it, you might be able to get away with that when using EPOLLONESHOT... – Voletta

So, multiple threads will epoll wait on one fd set? If you use level triggered they all wake up, but if you use edge triggered only one wakes up? – Doro

Yes, as the level remains "readable" for some (short amount of time until the data is actually read), all threads will be woken up. For edge-triggered on the other hand you only get a single state transition where a single thread is woken up. – Voletta

OK, thanks for the update. I'll modify my server to do threaded consumption of messages and see if there is a difference. – Doro

There is a significant difference, especially in outliers, if I use multiple threads to epoll_wait. For a trivially small (<100) file descriptors I saw no difference in ET or LT epoll using a single threaded consumer. – Doro

Actually there is no need for the "useless" EWOULDBLOCK recv syscall. If recv returns less than the max-read-size then you can be sure that at this point the kernel-buffer is empty and even if it fills back during your processing of the data epoll_wait notifies you again (in EPOLLET-mode). So the useless call is just needed in the unlikely case that the size of you recv-buffer is exact the length of the message. – Hammel

Concerning last point, read<max is only a hint, never a guarantee. We've had issues in the past with some kernel-side optimizations causing less than requested to be returned in a single recv() on a socket where data received with GRO were still pending. – Woolf

Answer 2

The difference is only visible when you use long-lived sessions and you're forced to constantly stop/start because of buffers full/empty (typically with a proxy). When you're doing this, you most often need an event cache, and when your event cache is processing events, you can use ET and avoid all the epoll_ctl(DEL)+epoll_ctl(ADD) dance. For short-lived sessions, the savings are less obvious, because for ET you'll need at least one epoll_ctl(ADD) call to enable polling on the FD, and if you don't expect to have more of them during the session's life (eg: exchanges are smaller than buffers most of the time), then you shouldn't expect any difference. Most of your savings will generally come from using an event cache only since you can often perform a lot of operations (eg: writes) without polling thanks to kernel buffers.

Woolf

Answer 3

When used as an edge-triggered interface, for performance reasons, it is possible to add the file descriptor inside the epoll interface (EPOLL_CTL_ADD) once by specifying (EPOLLIN|EPOLLOUT). This allows you to avoid continuously switching between EPOLLIN and EPOLLOUT calling epoll_ctl(2) with EPOLL_CTL_MOD.

Q9 Do I need to continuously read/write a file descriptor until EAGAIN when using the EPOLLET flag (edge-triggered behavior) ?

   A9  Receiving  an  event  from epoll_wait(2) should suggest to you that
       such file descriptor is ready for the requested I/O operation.  You
       must  consider  it  ready  until  the next (nonblocking) read/write
       yields EAGAIN.  When and how you will use the  file  descriptor  is
       entirely up to you.

       For packet/token-oriented files (e.g., datagram socket, terminal in
       canonical mode), the only way to detect the end of  the  read/write
       I/O space is to continue to read/write until EAGAIN.

       For  stream-oriented  files  (e.g., pipe, FIFO, stream socket), the
       condition that the read/write I/O space is exhausted  can  also  be
       detected  by checking the amount of data read from / written to the
       target file descriptor.  For example, if you call read(2) by asking
       to read a certain amount of data and read(2) returns a lower number
       of bytes, you can be sure of having exhausted the  read  I/O  space
       for  the  file  descriptor.   The  same  is true when writing using
       write(2).  (Avoid this latter technique  if  you  cannot  guarantee
       that  the  monitored file descriptor always refers to a stream-ori‐
       ented file.)

Pyrophotometer