How to append to a file-backed mmap using the memmap crate?

Question

I have a file foo.txt with the content

foobar

I want to continuously append to this file and have access to the modified file.

MmapMut

The first thing I tried is to mutate the mmap directly:

use memmap;
use std::fs;
use std::io::prelude::*;

fn main() -> Result<(), Box<std::error::Error>> {
    let backing_file = fs::OpenOptions::new()
        .read(true)
        .append(true)
        .create(true)
        .write(true)
        .open("foo.txt")?;

    let mut mmap = unsafe { memmap::MmapMut::map_mut(&backing_file)? };

    loop {
        println!("{}", std::str::from_utf8(&mmap[..])?);
        std::thread::sleep(std::time::Duration::from_secs(5));
        let buf = b"somestring";
        (&mut mmap[..]).write_all(buf)?;
        mmap.flush()?;
    }
}

This will result in a panic:

Error: Custom { kind: WriteZero, error: StringError("failed to write whole buffer") }

The resulting file reads somest

Appending to the backing file directly

After that, I tried to append to the backing file directly:

use memmap;
use std::fs;
use std::io::prelude::*;

fn main() -> Result<(), Box<std::error::Error>> {
    let mut backing_file = fs::OpenOptions::new()
        .read(true)
        .append(true)
        .create(true)
        .write(true)
        .open("foo.txt")?;

    let mmap = unsafe { memmap::MmapMut::map_mut(&backing_file)? };

    loop {
        println!("{}", std::str::from_utf8(&mmap[..])?);
        std::thread::sleep(std::time::Duration::from_secs(5));
        let buf = b"somestring";
        backing_file.write_all(buf)?;
        backing_file.flush()?;
    }
}

This does not result in a panic. The file will get updated regularly, but my mmap does not reflect these changes. I expected standard output to look like this:

foobar
foobarsomestring
foobarsomestringsomestring
...

But I got

foobar
foobar
foobar
...

I am mainly interested in a Linux solution if it is platform-dependent.

Bostow

That crate is already suspicious as every single access of an mmapped file is unsafe by Rust's terms. That type allows you to access the file while another process may be modifying it without any kind of guards. – Actinon

See this and this – Proboscis

Answer 1

First off, based on my understanding, I would urge you to be highly suspicious of that crate. It allows you do do things in safe Rust that you should not.

For example, if you have a file-backed mmap, then any process on your computer with the right permissions to the file can modify it. This means that:

1. It's never valid for the mmapped file to be treated as an immutable slice of bytes (&[u8]) because it might be mutated!
2. It's never valid for the mmapped file to be treated as a mutable slice of bytes (&mut [u8]) because a mutable reference implies an exclusive owner that can change that data, but you don't have that.

The documentation for that crate covers none of these concerns and doesn't discuss how you are supposed to use the small handful of unsafe functions in a safe manner. To me, these are signs that you may be introducing undefined behavior into your code, which is a Very Bad Thing.

For example:

use memmap;
use std::{fs, io::prelude::*};

fn main() -> Result<(), Box<std::error::Error>> {
    let mut backing_file = fs::OpenOptions::new()
        .read(true)
        .append(true)
        .create(true)
        .write(true)
        .open("foo.txt")?;

    backing_file.write_all(b"initial")?;

    let mut mmap_mut = unsafe { memmap::MmapMut::map_mut(&backing_file)? };
    let mmap_immut = unsafe { memmap::Mmap::map(&backing_file)? };

    // Code after here violates the rules of references, but doesn't use `unsafe`
    let a_str: &str = std::str::from_utf8(&mmap_immut)?;
    println!("{}", a_str); // initial

    mmap_mut[0] = b'x';

    // Look, we just changed an "immutable reference"!
    println!("{}", a_str); // xnitial

    Ok(())
}

---

Since people generally don't like being told "no, don't do that, it's a bad idea", here's how to get your code to "work": directly append to the file and then recreate the mmap:

use memmap;
use std::{fs, io::prelude::*, thread, time::Duration};

fn main() -> Result<(), Box<std::error::Error>> {
    let mut backing_file = fs::OpenOptions::new()
        .read(true)
        .append(true)
        .create(true)
        .write(true)
        .open("foo.txt")?;

    // mmap requires that the initial mapping be non-zero
    backing_file.write_all(b"initial")?;

    for _ in 0..3 {
        let mmap = unsafe { memmap::MmapMut::map_mut(&backing_file)? };

        // I think this line can introduce memory unsafety
        println!("{}", std::str::from_utf8(&mmap[..])?);

        thread::sleep(Duration::from_secs(1));

        backing_file.write_all(b"somestring")?;
    }

    Ok(())
}

You may want to preallocate a "big" chunk of space in this file so that you can just open it up and start writing, instead of having to re-map it.

I would not use this code for anything where it's important that the data is correct, myself.

See also:

• Is appending to a file that is Mmap-ed safe and cross-platform?
• appending to a memory-mapped file
• How to create and write to memory mapped files?

Actinon