How to create and write to memory mapped files?

Question

Editor's note: This code example is from a version of Rust prior to 1.0 and the code it uses does not exist in Rust 1.0. Some answers have been updated to answer the core question for newer versions of Rust.

I'm trying to create a memory mapped file using std::os::MemoryMap. The current approach looks as follows:

use std::os;
use std::ptr;
use std::old_io as io;
use std::os::unix::prelude::AsRawFd;
use std::os::MapOption;

let path = Path::new("test.mmap");

let f = match io::File::open_mode(&path, io::Open, io::ReadWrite) {
    Ok(f) => f,
    Err(err) => panic!("Could not open file: {}", err),
};

let mmap_opts = &[
    MapOption::MapReadable,
    MapOption::MapWritable,
    MapOption::MapFd(f.as_raw_fd())
];

let mmap = match os::MemoryMap::new(1024*1024, mmap_opts) {
    Ok(mmap) => {
        println!("Successfully created the mmap: {}", mmap.len());
        mmap
    }
    Err(err) => panic!("Could not read the mmap: {}", err),
};

unsafe {
   let data = mmap.data();

    if data.is_null() {
        panic!("Could not access data from memory mapped file")
    }

    let src = "Hello!";
    ptr::copy_memory(data, src.as_ptr(), src.as_bytes().len());
}

This program fails with

Process didn't exit successfully: `target/mmap` (status=4)

when calling ptr::copy_memory or any other operations on data.

• What is the reason I cannot write (or read) the data from the MemoryMap?
• What is the correct way to use MemoryMap in Rust?

Inseminate

What do you mean by "fails when calling..."? Is there a compiler error, a run-time crash, what's the error message, etc.? – Glans

Do you want the modifications to be saved back to the file? – Polaris

@delnan, updated the question with the error message – Inseminate

@Shepmaster, yes, that would be ideal and I saw that you included it in your answer, thanks a lot! – Inseminate

Answer 1

The real answer is to use a crate that provides this functionality, ideally in a cross-platform manner.

use memmap; // 0.7.0
use std::{
    fs::OpenOptions,
    io::{Seek, SeekFrom, Write},
};

const SIZE: u64 = 1024 * 1024;

fn main() {
    let src = "Hello!";

    let mut f = OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open("test.mmap")
        .expect("Unable to open file");

    // Allocate space in the file first
    f.seek(SeekFrom::Start(SIZE)).unwrap();
    f.write_all(&[0]).unwrap();
    f.seek(SeekFrom::Start(0)).unwrap();

    let mut data = unsafe {
        memmap::MmapOptions::new()
            .map_mut(&f)
            .expect("Could not access data from memory mapped file")
    };

    data[..src.len()].copy_from_slice(src.as_bytes());
}

Note that this is still possible for this code to lead to undefined behavior. Since the slice is backed by a file, the contents of the file (and thus the slice) may change from outside of the Rust program, breaking the invariants that the unsafe block is supposed to hold. The programmer needs to ensure that the file doesn't change during the lifetime of the map. Unfortunately, the crate itself does not provide much assistance to prevent this from happening or even any documentation warning the user.

---

If you wish to use lower-level system calls, you are missing two main parts:

1. mmap doesn't allocate any space on its own, so you need to set some space in the file. Without this, I get Illegal instruction: 4 when running on macOS.

2. MemoryMap (was) private by default so you need to mark the mapping as public so that changes are written back to the file (I'm assuming you want the writes to be saved). Without this, the code runs, but the file is never changed.

Here's a version that works for me:

use libc; // 0.2.67
use std::{
    fs::OpenOptions,
    io::{Seek, SeekFrom, Write},
    os::unix::prelude::AsRawFd,
    ptr,
};

fn main() {
    let src = "Hello!";

    let size = 1024 * 1024;

    let mut f = OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open("test.mmap")
        .expect("Unable to open file");

    // Allocate space in the file first
    f.seek(SeekFrom::Start(size as u64)).unwrap();
    f.write_all(&[0]).unwrap();
    f.seek(SeekFrom::Start(0)).unwrap();

    // This refers to the `File` but doesn't use lifetimes to indicate
    // that. This is very dangerous, and you need to be careful.
    unsafe {
        let data = libc::mmap(
            /* addr: */ ptr::null_mut(),
            /* len: */ size,
            /* prot: */ libc::PROT_READ | libc::PROT_WRITE,
            // Then make the mapping *public* so it is written back to the file
            /* flags: */ libc::MAP_SHARED,
            /* fd: */ f.as_raw_fd(),
            /* offset: */ 0,
        );

        if data == libc::MAP_FAILED {
            panic!("Could not access data from memory mapped file")
        }

        ptr::copy_nonoverlapping(src.as_ptr(), data as *mut u8, src.len());
    }
}

Polaris

I don't understand why file contents changing can cause undefined behavior here. Is it due to the memory model (non-volatile reads)? – Plunkett

@Plunkett It's declared that mutating an immutable value is undefined behavior. If the file changes, the mapped memory will change, regardless if the Rust program has marked it as immutable. – Polaris

Answer 2

Up to date version:

use std::ptr;
use std::fs;
use std::io::{Write, SeekFrom, Seek};
use std::os::unix::prelude::AsRawFd;
use mmap::{MemoryMap, MapOption};

// from crates.io
extern crate mmap;
extern crate libc;

fn main() {
    let size: usize = 1024*1024;

    let mut f = fs::OpenOptions::new().read(true)
                                      .write(true)
                                      .create(true)
                                      .open("test.mmap")
                                      .unwrap();

    // Allocate space in the file first
    f.seek(SeekFrom::Start(size as u64)).unwrap();
    f.write_all(&[0]).unwrap();
    f.seek(SeekFrom::Start(0)).unwrap();

    let mmap_opts = &[
        // Then make the mapping *public* so it is written back to the file
        MapOption::MapNonStandardFlags(libc::consts::os::posix88::MAP_SHARED),
        MapOption::MapReadable,
        MapOption::MapWritable,
        MapOption::MapFd(f.as_raw_fd()),
    ];

    let mmap = MemoryMap::new(size, mmap_opts).unwrap();

    let data = mmap.data();

    if data.is_null() {
        panic!("Could not access data from memory mapped file")
    }

    let src = "Hello!";
    let src_data = src.as_bytes();

    unsafe {
        ptr::copy(src_data.as_ptr(), data, src_data.len());
    }
}

Douglasdouglashome

I get error[E0433]: failed to resolve. Could not find 'consts' in 'libc' trying to compile. Fixed changing libc::consts::os::posix88::MAP_SHARED to libc::MAP_SHARED. – Authorization

Answer 3

2022-version:

use memmap2::Mmap;
use std::fs::{self};
use std::io::{Seek, SeekFrom, Write};
use std::ops::DerefMut;

pub fn memmap2() {
    // How to write to a file using mmap
    // First open the file with writing option
    let mut file = fs::OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open("mmap_write_example2.txt")
        .unwrap();

    // Allocate space in the file for the data to be written,
    // UTF8-encode string to get byte slice.
    let data_to_write: &[u8] =
        "Once upon a midnight dreary as I pondered weak and weary; äåößf\n".as_bytes();
    let size: usize = data_to_write.len();
    file.seek(SeekFrom::Start(size as u64 - 1)).unwrap();
    file.write_all(&[0]).unwrap();
    file.seek(SeekFrom::Start(0)).unwrap();

    // Then write to the file
    let mmap = unsafe { Mmap::map(&file).unwrap() };
    let mut mut_mmap = mmap.make_mut().unwrap();
    mut_mmap.deref_mut().write_all(data_to_write).unwrap();
}

Teofilateosinte

Note that the other answers are outdated and used libraries that haven't been been touched since 2015. – Champollion