read and write to same socket (TCP) using select

Question

We're writing a client and a server to do (what I thought was) pretty simple network communications. Mulitple clients connect to the server which then is supposed to send the data back to all other clients.

The server just sits in a blocking select loop waiting for traffic, and when it comes, sends the data to the other clients. This seems to work just fine.

The problem is the client. In response to a read, it will sometimes want to do a write.

However, I've found that if I use:

 rv = select(fdmax + 1, &master_list, NULL, NULL, NULL);

My code will block until there is new data to read. But sometimes (asynchronously, from another thread) I'll have new data to write on the network communication thread. So, I want my select to wake up periodically and let me check if there are data to write, like:

if (select(....) != -1)
{
  if (FD_SET(sockfd, &master_list))
     // handle data or disconnect
  else
     // look for data to write and write() / send() those.
}

I tried setting the select to poll mode (or ridiculously short timeouts) with:

// master list contains the sockfd from the getaddrinfo/socket/connect seq
struct timeval t;
memset(&t, 0, sizeof t);
rv = select(fdmax + 1, &master_list, NULL, NULL, &t);

but have found that then then client never gets any incoming data.

I also tried setting the socket fd to be non-blocking, like:

fcntl(sockfd, F_SETFL, O_NONBLOCK);

but that doesn't solve the problem:

1. if my client select() has no struct timeval, reading data works, but it never unblocks to let me look for writable data.
2. if my client select() has a timeval to get it to poll, then it never signals that there are incoming data to read, and my app freezes thinking there is no network connection made (despite the fact that all other function calls have succeeded)

Any pointers at all as to what I could be doing wrong? Is it not possible to do read-write on the same socket (I can't believe that to be true).

(EDIT: The correct answer, and thing that I remembered on the server but not on the client, is to have a second fd_set, and copy master_list before each call to select():

// declare and FD_ZERO read_fds:
// put sockfd in master_list

while (1)
{
   read_fds = master_list;
   select(...);

   if (FD_ISSET(read_fds))
     ....
   else
     // sleep or otherwise don't hog cpu resources
}

)

Feticide

sorry, that if (FD_SET( ... should be if (FD_ISSET( .... – Feticide

Answer 1

Everything looks fine, except the line where you do if (FD_SET(sockfd, &master_list)). I have a very similar code structure and I used FD_ISSET. You're supposed to test if the list is set, not to set it again. Other than that, I see nothing else.

Edit. Also, I have the following for the timeout:

timeval listening_timeout;
listening_timeout.tv_sec = timeout_in_seconds;
listening_timeout.tv_usec = 0;

perhaps there's an issue if you set it to 0 (as you seem to be doing?)

Edit2. I remembered I ran into a strange problem when I wasn't clearing the read set after the select exited and before I entered it again. I had to do something like:

FD_ZERO(&sockfd);
FD_SET(sockfd, &rd);

before I was entering select. I can't remember why though.

Augustineaugustinian

There shouldn't be any problem with setting the timeout to 0: in that case select will just check whether any descriptors are currently ready and return immediately. – Bruxelles

Re your Edit2: select will modify the sets, so you need to reset them before calling it again. – Bruxelles

@jk thanks, I'll add that as a comment to my source file, it will be useful in the future – Augustineaugustinian

Actually, @jk, that little comment you threw in there about resetting the fd_set is it. You are the winrars!!11!one! – Feticide

Answer 2

I seem to recall a trick about creating and sharing a read/write filedescriptor between the network thread and the main thread that is added to the descriptors in the select call. This fd has one byte written to it by the main thread when it has something to send. The write wakes up network thread from the select call and the network thread is then grabs the data from a shared buffer and writes it to the network then go back to sleep in the select.

Sorry if that's a bit vague and lacking of code ... and my memory may be incorrect.. so others may have to guide you further.

Profluent

I really like this idea a lot. Even though it doesn't solve my problem, I already use a pipe to tell the network thread to exit, and hadn't made the leap to also using to tell the thread that there are data available to write. I love it !! – Feticide

Answer 3

I don't see anything wrong with your code, so it should work. If you can't get it working, one way to work around it would be to create a pipe to be used by your reading thread and the thread that prepares things for writing, and add the pipe's reading end to your select set. Then, when the other thread has prepared data to write, it just sends something on the pipe, your reading thread gets woken up from the select, and it can then do the writing. Depending on how often there is data to read or write, this could also be more efficient.

Bruxelles

Well, the code up top was indeed basically correct (except the FD_SET / FD_ISSET thing). What was not correct was what was missing, and what @jk pointed out above — you have to reset the fd_set before each call to select. I created a new one called read_fds and just before each call to select "read_fds = master_list", and then all subsequent FD_ISSET calls look in read_fds, not master_list. Thanks for that. Read/write sockets working again, wooO! – Feticide

It was actually @Augustineaugustinian who pointed that out, I just explained the reason. I would have accepted her answer, as it's the one that actually solved the problem. – Bruxelles

Answer 4

2 threads should be able to work with the same socket at a time, so your main thread should be able to write to a client while the other one sleeps in select waiting for incoming data. This of course assumes that both threads have access to the client list.

Jeffreys