epoll_wait on several Threads faster?

Question

I am thinking of programming a tcp server based on epoll. To achieve the best performance i want to implement multi core support too. But during my researches the following question came up: Is it faster to call two epoll_wait()-Calls from two different threads, each observing their own file descriptors on a dual core? Or is this as fast as calling just one single epoll_wait() which observes all file descriptors?

Since the Kernel observes the file descriptors, i think it doens't matter how much threads i use on user space calling epoll_wait()?

Pleistocene

How often do you expect changes to occur? – Surmullet

Answer 1

You can even call epoll_wait concurrently on multiple threads for the same epoll_fd as long as you use edge-triggered (EPOLLET) mode (and be careful about synchronisation). Using that approach you can get real performance benefits on multi-core machines compared to a single-threaded epoll event loop.

I have actually done performance measurements some time ago, see my blog postings for the results:

• http://cmeerw.org/blog/748.html#748
• http://cmeerw.org/blog/746.html#746

Ineducation

Why only with EPOLLET mode? – Inextirpable

Because if you work in any other mode, then the epoll will report the same file descriptors to several threads, causing spurious wake ups. – Vallecula

This link (idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12) can help with understanding the "spurious wake ups" which the link refers to as thundering herd. – Alpenstock

Answer 2

I suspect you're correct that the performance of epoll per se is no different in the two scenarios you mention. What could make a difference, OTOH, is that by having an event loop in each thread you don't need to context switch to a separate worker thread to handle the connection. This is e.g. how nginx works, see e.g. http://www.aosabook.org/en/nginx.html .

Wendelina