What's the deal with boost.asio and file i/o?

Question

I've noticed that boost.asio has a lot of examples involving sockets, serial ports, and all sorts of non-file examples. Google hasn't really turned up a lot for me that mentions if asio is a good or valid approach for doing asynchronous file i/o.

I've got gobs of data i'd like to write to disk asynchronously. This can be done with native overlapped io in Windows (my platform), but I'd prefer to have a platform independent solution.

I'm curious if

1. boost.asio has any kind of file support
2. boost.asio file support is mature enough for everyday file i/o
3. Will file support ever be added? Whats the outlook for this?

Fonzie

The general idea is that files should be read-at-hand the majority of the time. Where sockets are often located thousands of miles apart and sometimes never complete. If you need asynchronous files that's usually because you are creating a GUI interface. Handle that by having a worker-thread process the blocking file I/O in the background. – Cavalcade

Related: #59668277 – Plumbism

Answer 1

Has boost.asio any kind of file support?

Starting with (I think) Boost 1.36 (which contains Asio 1.2.0) you can use [boost::asio::]windows::stream_handle or windows::random_access_handle to wrap a HANDLE and perform asynchronous read and write methods on it that use the OVERLAPPED structure internally.

User Lazin also mentions boost::asio::windows::random_access_handle that can be used for async operations (e.g. named pipes, but also files).

Is boost.asio file support mature enough for everyday file i/o?

As Boost.Asio in itself is widely used by now, and the implementation uses overlapped IO internally, I would say yes.

Will file support ever be added? Whats the outlook for this?

As there's no roadmap found on the Asio website, I would say that there will be no new additions to Boost.Asio for this feature. Although there's always the chance of contributors adding code and classes to Boost.Asio. Maybe you can even contribute the missing parts yourself! :-)

Adiathermancy

Thanks, that may prove useful. Do you know of any plans to implement a platform independent asynchronous file? – Fonzie

Not that I know of, sorry. There is a posix::stream_descriptor class that does the same for posix handles, so at least a wrapper for the two approaches could be written. – Adiathermancy

I believe that the posix::stream_descriptor will not work with regular files. See boost.org/doc/libs/1_43_0/doc/html/boost_asio/overview/posix/…. – Hurdygurdy

I also read that on the boost.asio.users list some time ago; seems there's no way with Boost.Asio... thread.gmane.org/gmane.comp.lib.boost.asio.user/4043 – Adiathermancy

File support has been added https://mcmap.net/q/261579/-what-39-s-the-deal-with-boost-asio-and-file-i-o – Buckhound

Answer 2

boost::asio file i/o on Linux

On Linux, asio uses the epoll mechanism to detect if a socket/file descriptor is ready for reading/writing. If you attempt to use vanilla asio on a regular file on Linux you'll get an "operation not permitted" exception because epoll does not support regular files on Linux.

The workaround is to configure asio to use the select mechanism on Linux. You can do this by defining BOOST_ASIO_DISABLE_EPOLL. The trade-off here being select tends to be slower than epoll if you're working with a large number of open sockets. Open a file regularly using open() and then pass the file descriptor to a boost::asio::posix::stream_descriptor.

boost::asio file i/o on Windows

On Windows you can use boost::asio::windows::object_handle to wrap a Handle that was created from a file operation. See example.

Ribose

On FreeBSD asio uses kqueue to multiplex system objects by default – Contrecoup

select will not do anything useful according to both the bugreport you listed "nothing meaningful is returned from poll/select on file system files." and this comment " Regular files are always readable and they are also always writeable. This is clearly stated in the relevant POSIX specifications. I cannot stress this enough. Putting a regular file in non-blocking has ABSOLUTELY no effects other than changing one bit in the file flags." – Strick

Answer 3

io_uring has changed everything.
asio now support async file read/write.
See the releases notes:
asio 1.21.0 releases notes

Lesser

Do you know if asio with uring supports writing multiple buffers ? liburing itself does, but not sure about the asio implementation – Drumfire

Answer 4

boost::asio::windows::random_access_handle is the easiest way to do this, if you need something advanced, for example asynchronous LockFileEx or something else, you might extend asio, add your own asynchronous events. example

Gillett

Answer 5

ASIO supports overlapped I/O on Windows where support is good. On Unixes this idea has stagnated due to:

• Files are often located on the same physical device, accessing them sequentially is preferable.
• File requests often complete very rapidly because they are physically closeby.
• Files are often critical to complete the basic operation of a program (e.g. reading in its configuration file must be done before initializing further)

The one common exception is serving files directly to sockets. This is such a common special-case that Linux has a kernel function that handles this for you. Again, negating the reason to use asynchronous file I/O.

In Short: ASIO appears to reflect the underlying OS design philosophy, overlapped I/O being ignored by most Unix developers, so it is not supported on that platform.

Cavalcade

Out of curiosity, what kernel function are you referring to? – Dode

Never mind, i believe it's sendfile which is also capable of sending to sockets: man7.org/linux/man-pages/man2/sendfile.2.html – Dode

The problem that needs to be addressed is: not allocating a thread stack for each pending file transfer. How does sendfile address this problem? Given that 99% of i/o for a web server is going to be static content, throttled at the client end, isn't that a pretty big problem? – Watchcase

Answer 6

Asio 1.21 appears to have added built-in filesystem support.

For instance, asio::stream_file now exists with all the async methods you'd expect.

Buckhound

Answer 7

Linux has an asio Library that is no harder to use than Windows APIs for this job (I've used it). Both sets of operating systems implement the same conceptual architecture. They differ in details that are relevant to writing a good library, but not to the point that you cannot have a common interface for both OS platforms (I've used one).

Basically, all flavors of Async File I/O follow the "Fry Cook" architecture. Here's what I mean in the context of a Read op: I (processing thread) go up to a fast food counter (OS) and ask for a cheeseburger (some data). It gives me a copy of my order ticket (some data structure) and issues a ticket in the back to the cook (the Kernel & file system) to cook my burger. I then go sit down or read my phone (do other work). Later, somebody announces that my burger is ready (a signal to the processing thread) and I collect my food (the read buffer).

Azobenzene

And who, exactly, is the fry cook on a LINUX asio web server? – Watchcase