Posix shared memory resize with mremap when several processes are using the segment

Question

I’m storing some data in a shared memory array used by several processes. At some point I want to grow the array.

Assume there’s already a synchronisation mechanism between the processes

Initially Process 1 will create the segment and process 2 will open it.

Process 1
shm_open() O_CREAT
ftruncate()
mmap() MAP_SHARED

Process 2
shm_open()
mmap()

At some point one process wants to grow the array and resize the shared segment.

Process 1 calls
ftruncate()
mremap() MREMAP_MAYMOVE

Shall Process 2 be notified of the resize and call mremap() to update it’s own virtual address too ?

If Process 2 has to be notified I’m thinking of opening a second shared memory segment with some metadata e.g the table’s capacity and a mutex. Each process stores the table’s capacity initially from shared memory and on each operation checks the local value against the shared memory metadata value. If the value has changed it will call mremap()

Is this a proper way to do this if mremap() has to be called on each process after resizing?

Botti

I believe the target process should update its own memory mapping on demand (when it wants to access something outside its bounds), but it does have to keep track of how much is the current mapping. But this option will break down if the array may shrink. If not though, this is lockfree. – Anandrous

Inasmuch as this question focuses on mremap(), which is Linux-specific, [linux] is a more applicable tag than [posix] (retagged). – Murder

Has any progress been made with respect to this problem? – Bumpy

Answer 1

Shall Process 2 be notified of the resize and call mremap() to update it’s own virtual address too ?

Process 2 has mapped a specific region of the shared memory segment. Another process increasing the size of the segment does not invalidate that mapping. Nor does that change the region of the shared memory segment that is mapped in Process 2 -- even if process 2 originally mapped the whole segment, the part beyond the original end of the segment is not automatically mapped in process 2.

Therefore, as long as process 2 does not require access to additional pages of the segment, it has no need to update its mapping at all. If it wants to access that additional shared memory, however, then it does need to update its mapping. It can attempt to do that via Linux-specific mremap(), or by munmap() followed by mmap(). The latter requires that it still have an open file descriptor for the segment. Either way, it might not be possible to map a larger space starting at the same base address. For that matter, it might not be possible to map a larger space at all, regardless of whether any other process was able to do so.

About the mapping's base address

By default, mremap() will attempt to modify the mapped region without changing its base address, and munmap() + mmap() can be asked to do the same by requesting the original base address from mmap() and passing the MAP_FIXED flag. These will fail if the mapping cannot be expanded at that address. In the latter case, that will also leave the segment altogether unmapped.

You can allow for a new base address to be chosen by specifying the MREMAP_MAYMOVE among the mremap() flags, or by avoiding specifying MAP_FIXED to mmap(). Such attempts may succeed when expanding the mapping at the same address fails, but note well that a change of base address invalidates all that process's pointers into the original mapping, wherever stored.

Therefore, if you're going to provide for a change of base address, then you're probably best off not storing any pointers into the mapping at all, except for a single one to the base address. In that case, access the contents via that pointer or otherwise relative to it.

If Process 2 has to be notified I’m thinking of opening a second shared memory segment with some metadata e.g the table’s capacity and a mutex. Each process stores the table’s capacity initially from shared memory and on each operation checks the local value against the shared memory metadata value. If the value has changed it will call mremap()

Is this a proper way to do this if mremap() has to be called on each process after resizing?

Provided that you properly synchronize access to the metadata, your scheme sounds feasible. In fact, although there may be other reasons to do so, nothing presented in the question suggests that it should even be necessary to put the metadata in a separate shared memory segment. The expansion should not affect the content of the original pages of the segment or their mapping in process 2, so if the metadata are stored there then process 2 should still be able to access them after the expansion.

Note, however, that if the mutex, semaphore, or similar used to synchronize access to the segment is inside that segment, then you need to account for the fact that mremap() may move it, and munmap() / mmap() will leave you in need of a more complicated recovery scheme for the case that the munmap() succeeds but the subsequent mmap() fails.

---

In understanding all this, it may be useful to keep in mind that

• the size of the shared memory segment is a property of the segment, maintained by the kernel.

• the characteristics of each mapping of the segment, including the region mapped (a specific range of pages) and the base address to which it is mapped, are properties of a specific process, independent of all other mappings of the segment in the same or other processes.

• the characteristics of each mapping of the segment are also largely independent of the segment itself. In particular, the the offset and size of the mapped region does not change in response to changes to the size of the segment.

Murder

I would clarify that using mremap() might change the base virtual address of the mapping inside the process, invalidating pointers that the process itself might have stored somewhere. In fact, even if we try to use munmap()+mmap() with MAP_FIXED and the original base address, the call might fail and errno will be set to EINVAL. See the man page of mmap for additional information. – Endoskeleton

Thanks, @SRG, I have added a section discussing those issues under the heading "About the mapping's base address". – Murder

Perfect @JohnBollinger, this is excellent. I have tried to give you the +50 bounty but apparently I can't until tomorrow, haha :). But I will do it, this answer is superb, thank you very much. – Endoskeleton

Answer 2

If Process 2 has to be notified I’m thinking of opening a second shared memory segment with some metadata e.g the table’s capacity and a mutex. Each process stores the table’s capacity initially from shared memory and on each operation checks the local value against the shared memory metadata value. If the value has changed it will call mremap()

Adding to John Bollinger's great answer, I would point out that there is a way not to have a shared memory segment with metadata. For instance, if you have the file descriptor that shm_open() provided you, an alternative solution is to force your other processes to check the size of the segment with fstat:

...

struct stat statbuf = { 0 };
int fd = shm_open(...);

...

// Check whether the shared segment has changed
fstat(fd, &statbuf);

if (statbuf.st_size != current_size) // current_size is stored somewhere
{
    // Remap the shared memory segment using statbuf.st_size here
}

···

You must consider the shared memory segment as a file somewhere in the system. For reference purposes, the shm_overview page suggests the use of fstat() for this purpose.

Endoskeleton