Getting number of bytes available to read in a socket

Question

Here's my scenario. I have a TCP client that is talking to the server. Both the server and the client are running on local machine (Windows).

The dialog goes something like:

1. Client sends data to the server (Request)
2. Client does shutdown for send on the socket
3. Client blocks via a read call for response
4. Server receives the data, processes, and sends back a response (one shot, not broken into chunks)
5. Server does shutdown for send on the socket
6. Client receives the response, and continues processing.

At step 3, I am using recv() call to block and read data from the socket. At this point, I would like to peek to see how many bytes of data is available, so that I can allocate so much memory. By design, it is known that the server has sent all the data, and there is no more data for this request to be sent. (See step 5 above).

I have tried recv() with MSG_PEEK option, but that does not seem to give the total number of bytes available.

Is there a way to retrieve it?

Thanks in advance.

Tycho

Just because the server has sent all the data, this does not mean it will all arrive at the same time. Just because recv() returns does not mean you have got everything. You should keep calling recv() until in indicates there is nothing left by returning 0 – Huckaback

I speak from ignorance here, but I suspect the answer is no, because there is no way for the kernel to know the total number of bytes until it has received all the packets. And that might not happen until after the client asks. So the best you can hope would be a lower limit, which is what MSG_PEEK gives you. – Protochordate

Answer 1

On Windows at least, you can use ioctlsocket() with the FIONREAD command to determine the current number of bytes that are available for recv() to read without blocking. By the time you actually call recv(), more bytes may have arrived, though.

As @LokiAstari said, you should be calling recv() in a loop until it returns 0 bytes to indicate the socket has been closed. You do not need to know how many bytes are available, just pass a fixed-length buffer each time, and recv() will return how many bytes were actually read. Append each non-zero-length buffer read into another buffer that grows as needed until you have received all of the data, and then process that second buffer when ready.

Harrod

Answer 2

Using TCP there is no way of knowing how much bytes there is. Remember that TCP is a "streaming" protocol, there is a start of the stream but no end until the connection has been closed.

What you can do is read small chunks into a buffer, and enlarge the buffer when needed.

Certes

What if the size of the buffer is more than than the bytes available and, hence, we are trying to read more bytes than the socket provides. – Catcher

@sumitdhyani Then you will get what is available, unless you use recv with the MSG_WAITALL in which case the call will block until the system has received enough data to fulfill your request. – Certes

but what if the connection has been closed, recv returned 0, and I just need to know where the received bytes end in my buffer? Where to put the \0 character to terminate the received string, because my buffer might contain random bytes. – Trichromatism

@Trichromatism If recv returns 0 then it hasn't written anything to your buffer, the contents of your buffer is unmodified since last successful call to recv. So for you the solution is to set string null-terminator after every call to recv where it returns a positive value. – Certes

Answer 3

One way to do this, without pre-allocating or re-allocating buffers is to: (a) send the number of bytes the sender is sending (e.g., as a four byte int, in network byte order), (b) have the receiver receive those four bytes and allocate a receive buffer, and then (c) have the receiver send the actual data of the previously sent length. Note that (c) can occur by the sender without receiving feedback or a reply regarding (a) from the receiver.

I had been trying to use fstat to get the amount of data available on a socket, but it does not appear to be portable. On a Mac system, it works for my application. On a Linux system, it does not.

Llywellyn

Answer 4

It partly depends on the size of the response because even if the send it as one chunk, the underlying transport may break it into chunks. For example, Ethernet has a maximum packet size of 1500 bytes. Although you send it all once, while it is likely that you will receive it all at once, it is not guaranteed if you are using TCP because it is stream oriented. You are much more likely to receive the response as a single packet if you use UDP.

Scots