Platform independent memory mapped [file] IO

Question

I've spent some time investigating memory mapped IO for an application I'm working on. I have some very large (TB scale) files, and I want to map segments from them into memory, for both reading and writing, making maximum use of OS-level caching. The software I'm writing needs to work under Unix/Linux and Windows... performance is critical.

I've discovered boost::iostreams::mapped_file_source and boost::iostreams::mapped_file_sink, which provide most of the facilities I'm looking for. The facilities I'd like, but haven't found are:

• Forcing a synchronisation of written data to disk ( msync(2) on Unix; FlushViewOfFile on Windows)
• Locking of files to prevent two processes attempting to write the same file at the same time (or read while the file is still being written..)
• Controlling attributes of the file at creation time (Unix)

Can I do these things using "boost/iostreams/device/mapped_file.hpp"? Are there other platform independent libraries that would better suit my requirements? Must I develop my own cross-platform library to get this flexibility?

Cristacristabel

BTW, "memory mapped I/O" generally refers to reading I/O ports using addresses (a.k.a. memory mapped) (like using a pointer) rather than using special processor I/O instructions. – Stephan

Fair point. I'm talking about memory mapped file I/O [ en.wikipedia.org/wiki/Memory-mapped_file ] - and have edited the title to reflect this. – Cristacristabel

Answer 1

Look at boost::interprocess and boost::interprocess::file_mapping. They have everything you need.

boost::interprocess

boost::interprocess::file_mapping

Propinquity

You might be right... but, with boost::interprocess, I can't see how I can (straightforwardly) establish access the data in OS-page-sized chunks... – Cristacristabel

@aSteve, this library supports memory mapped files, see: boost.org/doc/libs/1_48_0/doc/html/interprocess/… – Propinquity

Yes, boost::interprocess supports (managed) memory mapped files... "managed" in this context implies that space in the file is accessed "as if" by malloc(). Obviously valuable for inter-process-communication, but I require a more basic "unmanaged?" approach for "raw" IO. I need a function that returns a pointer to a mapped block containing an arbitrary file offset. The boost::iostreams::mapped_file_* interfaces are ideal here... though they don't offer the same flexibility for synchronisation/locking. I've read the interprocess docs - I can't see how to use interprocess for "raw" mapped IO. – Cristacristabel

Update... I now realise you meant that I should use boost::interprocess::file_mapping - and this class makes perfect sense. – Cristacristabel

@aSteve, sorry, I thought you'd explore the library once you were aware of it! :) it seems you have now! ;) – Propinquity

Answer 2

This question is pretty old but the top hit on google. So I'll add some other libs I know of.

mio Cross-platform C++11 header-only library for memory mapped file IO.

• Seems simple and straightforward, but potentially abandoned.

llfio P1031 low level file i/o and filesystem library for the C++ standard.

• The mother of all io libraries it seems. Has gone through boost peer review (aka torture). Is proposed as an addition to C++ standard library.

I can also vouch for boost::interprocess, I've used it to great effect in the past.

Dropforge