appending to a memory-mapped file

Question

I'm constantly appending to a file of stock quotes (ints, longs, doubles, etc.). I have this file mapped into memory with mmap.

What's the most efficient way to make newly appended data available as part of the memory mapping?

I understand that I can open the file again (new file descriptor) and then mmap it to get the new data but that seems to be inefficient. Another approach that has been suggested to me is to pre-allocate the file in 1mb chunks, write to a specific position until reaching the end then ftruncate the file to +1mb.

Are there other approaches?

Doest Boost help with this?

Marlomarlon

Answer 1

Boost.IOStreams has fixed-size only memory mapped files, so it won't help with your specific problem. Linux has an interface mremap which works as follows:

void *new_mapping = mremap(mapping, size, size + GROWTH, MREMAP_MAYMOVE);
if (new_mapping == MAP_FAILED)
    // handle error
mapping = new_mapping;

This is non-portable, however (and poorly documented). Mac OS X seems not to have mremap.

In any case, you don't need to reopen the file, just munmap it and mmap it again:

void *append(int fd, char const *data, size_t nbytes, void *map, size_t &len)
{
    // TODO: check for errors here!
    ssize_t written = write(fd, data, nbytes);
    munmap(map, len);
    len += written;
    return mmap(NULL, len, PROT_READ, 0, fd, 0);
}

A pre-allocation scheme may be very useful here. Be sure to keep track of the file's actual length and truncate it once more before closing.

Mohandas

Answer 2

I know the answer has already been accepted but maybe it will help someone else if I provide my answer. Allocate a large file ahead of time, say 10 GiB in size. Create three of these files ahead of time, I call them volumes. Keep track of your last known location somewhere like in the header, another file, etc. and then keep appending from that point. If you reach the maximum size of the file and run out of room switch to the next volume. If there are no more volumes, create another volume. Note that you would probably do this a few volumes ahead to make sure not to block your appends waiting for a new volume to be created. That's how we implement it where I work for storing continuous incoming video/audio in a DVR system for surveillance. We don't waste space to store file names for video clips which is why we don't use a real file system and instead we go flat file and we just track offsets, frame information (fps, frame type, width/height, etc), time recorded and camera channel. For you storage space is cheap for the kind of work you are doing, whereas your time is invaluable. So, grab as much as you want to ahead of time. You're basically implementing your own file system optimized for your needs. The needs that general-use file systems supply aren't the same needs that we need in other fields.

Letter

Answer 3

Looking at man page for mremap it should be possible.

V2

mremap is Linux-specific, though. – Marlomarlon

Answer 4

My 5cents, but they are more C specific. Make normal file, but mmap huge size - e.g file is say 100K, but mmap 1GB or more. Then you can safely access everything up to file size. Access over file size will result in error. If you are on 32bit OS, just dont make mmap too big, because it will eat your address space.

Exobiology

Answer 5

If you're using boost/iostreams/device/mapped_file.hpp on windows:

boost::filesystem::resize_file throws an exception if a reading mapping object is open, due to lack of sharing privileges. Instead, use windows-api to resize the file on the disc, and the reading mapped_files can still be open.

bool resize_file_wapi(string path, __int64 new_file_size) //boost::uintmax_t size
{
    HANDLE handle = CreateFile(path.c_str(), GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, 0, OPEN_EXISTING,
    FILE_ATTRIBUTE_NORMAL, 0);
    LARGE_INTEGER sz;
    sz.QuadPart = new_file_size;

    return handle != INVALID_HANDLE_VALUE
    && ::SetFilePointerEx(handle, sz, 0, FILE_BEGIN)
    && ::SetEndOfFile(handle)
    && ::CloseHandle(handle);
}

Metastasis