Cross-platform (linux/Win32) nonblocking C++ IO on stdin/stdout/stderr

Question

I'm trying to find the best solution for nonblocking IO via stdin/stdout with the following characteristics:

• As long as there is enough data, read in n-sized chunks.
• If there's not enough data, read in a partial chunk.
• If there is no data available, block until there is some (even though it may be smaller than n).

The goal is to allow efficient transfer for large datasets while processing 'control' codes immediately (instead of having them linger in some partially-filled buffer somewhere).

I know I can achieve this by using threads and a istream::get() loop, or by writing a bunch of platform-specific code (since you can't select() on file handles in windows)... ((There is also istream::readsome() which seems promising, but the only results I can find on google were of people saying it doesn't actually work well.))

Since I haven't done much coding w/ these APIs, perhaps there is a better way.

Bowknot

Answer 1

Maybe boost::asio can be of use for you?

Mousetail

That looks rather promising (a big hammer for a small problem, but probably worth learning)... Thanks! – Bowknot

Yep, that's probably your best bet for portable asynchronous I/O, since the language doesn't have native support for nonblocking I/O or threads. You'll have to use a library of some kind, and boost is usually a good bet. – Boonie

(If only Win32 supported select() or poll() on arbitrary handles instead of just sockets)... After doing some digging, I've confirmed that I can do what I need and there's even a tutorial for setting it up: highscore.de/boost/process/process/… – Bowknot

Addendum: The tutorial is for snagging the stdin/stdout of a child process using async streams; what I want is for my own process to deal with its own stdin/stout in an async manner. Still, this is a nice 90% of the way there. – Bowknot

@DanS.: Did you ever find a 100% solution? I am now trying to solve the same problem – Columella

Answer 2

I used the threads and platform specific code. See my answer to another question. I was able to put the OS-specific stuff in inputAvailable() (Linux uses select, Windows just returns true). I could then use WaitForSingleObject() with a timeout on Windows to try to let the thread complete, then TerminateThread() to kill it. Very ugly, but the team didn't want to use this bit of boost.

Noteworthy

This only links to a non cross platform solution though (also includes a few stale links). Do you have the original implementation available somewhere? – Tradespeople

Sorry, that was a past company, I no longer have that code. I believe the code for inputAvailable() is the Linux solution. Check out _kbit() as a starting place for Windows: learn.microsoft.com/en-us/cpp/c-runtime-library/reference/… – Noteworthy

Answer 3

I did something similar to jwhitlock ... I ended up with a StdinDataIO class that wraps around the appropriate OS-specific implementation(*) so that the rest of my program can select() on the file descriptor StdinDataIO provides, remaining blissfully ignorant of Windows' limitations regarding stdin. Have a look here and here if you like, the code is all open-source/BSD-licensed.

(*) the implementation is a simple pass-through for Linux/MacOSX, and in Windows it's a rather complex process of setting up a child thread to read from stdin and send the data it receives over a socket back to the main thread... not very elegant, but it works.

Bohemia