Question about epoll and splice

Question

My application is going to send huge amount of data over network, so I decided (because I'm using Linux) to use epoll and splice. Here's how I see it (pseudocode):

epoll_ctl (file_fd, EPOLL_CTL_ADD); // waiting for EPOLLIN event
while(1)
{
    epoll_wait (tmp_structure);

    if (tmp_structure->fd == file_descriptor)
    {
        epoll_ctl (file_fd, EPOLL_CTL_DEL); 
        epoll_ctl (tcp_socket_fd, EPOLL_CTL_ADD); // wait for EPOLLOUT event
    }

    if (tmp_structure->fd == tcp_socket_descriptor)
    {
        splice (file_fd, tcp_socket_fd);
        epoll_ctl (tcp_socket_fd, EPOLL_CTL_DEL); 
        epoll_ctl (file_fd, EPOLL_CTL_ADD); // waiting for EPOLLIN event
    }
}

I assume, that my application will open up to 2000 TCP sockets. I want o ask you about two things:

1. There will be quite a lot of epoll_ctl calls, won't wit be slow when I will have so many sockets?
2. File descriptor has to become readable first and there will be some interval before socket will become writable. Can I be sure, that at the moment when socket becomes writable file descriptor is still readable (to avoid blocking call)?

Elwina

Answer 1

1st question

1. You can use edge triggered rather then even triggered polling thus you do not have to delete socket each time.
2. You can use EPOLLONESHOT to prevent removing socket

File descriptor has to become readable first and there will be some interval before socket will become writable.

What kind of file descriptor? If this file on file system you can't use select/poll or other tools for this purpose, file will be always readable or writeable regardless the state if disk and cache. If you need to do staff asynchronous you may use aio_* API but generally just read from file write to file and assume it is non-blocking.

If it is TCP socket then it would be writeable most of the time. It is better to use non-blocking calls and put sockets to epoll when you get EWOULDBLOCK.

Rube

By 'file_descriptor' I mean descriptor of disk file opened with open() syscall, so as I see my idea cloud work just fine :) – Elwina

@Goofy, you can't use select/epoll with disks on file as they would always report readability/writability even if operation can block to bring data from disk. See kegel.com/c10k.html – Rube

Answer 2

Consider using EPOLLET flag. This is definitely for that case. When using this flag you can use event loop in a proper way without deregistering (or modifying mode on) file descriptors since first registration in epoll. :) enjoy!

Malorie