Is there a way to an share address mapping between two unrelated processes on Linux?

Question

I'm assuming if this is possible it would most likely be via mmap

My original question was can mmap() return the same pointer for each user of the same piece of shared memory? Ultimately an address in each processes address space maps to the same physical memory. The question is whether the address provided to each process can be made the same somehow.

---

Why does this matter?

Consider the case of sharing a data structure which contains pointers. If the address mapping is shared those pointers are valid for both processes. If the address mapping is not shared then a pointer may point to the address space for the process that wrote it. If the other process tries to read it, it will trigger an access violation.

For shared libraries we rely on position independent code but there can still be pointers that the loader needs to adjust on load. (I am unclear whether this is done exactly once or whether each process gets its own private copy of pages containing such pointers adjusted to fit its own address space).

If the address mapping is not shared then for data we either need to create position independent structures (unfortunate acronym) using offsets instead of absolute pointers or limit ourselves to data structures not containing any pointers.

In my application I have several data structures I wish to share which are essentially vectors. I do not wish to persist or serialise/deserialise them. I want the same physical memory to be used by each process. Basically its a search problem - the shared data is the haystack and each process searches for its own set of needles. For various reasons we do not want them to be threads though it would eliminate this question entirely because threads do share the address space.

I can create vectors inside shared memory using a custom allocator. Say:

SharedMemoryAllocator sharedAlloc(somePointerFromMmap);
class Foo
{
   std::vector<Bar, SharedMemoryAllocator>   A;
   std::vector<Snafu, SharedMemoryAllocator> B;

   Foo(SharedMemoryAllocator& sharedAlloc):
      A(sharedAlloc)
      B(sharedAlloc)
   {
   }
};

If the address mapping is shared it is safe to use such an allocator for the complete structure:

unique_ptr<Foo> fubar = sharedAlloc.new(Foo(sharedAlloc));

Because the pointers to storage for A and B with *fubar will be in the same address space for both.

If on the other hand the address mapping is not shared I can only safely share the underlying memory blocks to which A and B point.

That is each process must have its own local instance of Foo where each vector is constructed by attaching to a shared memory block provided by the allocator. This is more portable but uglier.

Technically neither conforms to the C++ spec without the addition of std::bless() and other wizardry. So we are in undefined (or rather platform defined) behaviour anyway. But then so is 'safe' use of malloc (see link above).

---

As far as I know nothing about the address returned by mmap() is guaranteed by Posix or Linux except for the case where you ask for a specific address and the call does not fail.

So if it happens to work it is probably at best undefined behaviour. Its generally a bad idea to rely on undefined behaviour for all the usual reasons. But perhaps it is actually platform defined rather than undefined?

As mentioned the exception is for a fixed address mapping but how can you get two independent processes to agree on the same safe address to use in advance? One possibility is given here

This suggests you need a second shared memory segment or some other IPC mechanism to share the address assigned to the creator of the shared memory block you wish to use.

Is this the correct way? Is it the only way?

---

Looking at When would one use mmap MAP_FIXED? - the main legitimate use of MAP_FIXED is to remap different kinds of memory segments which need to be at the same relative address to each other when loading a library

However other people are using this the way I suggest. The other answer to that question mentions:

• Shared memory may contain pointers.

I found a few other references to people using MAP_FIXED to do this but I have not yet found a working example.

Has use of MAP_FIXED this way been rendered non-functional by ASLR ?

It is actually the case that you can only share address space in very specific circumstances:

For example only if:

• You use MAP_ANONYMOUS and

• pass the FD to an unrelated process via sendmsg/recvmsg

  Apparently linux does not support I_SENDFD/I_RECVFD

Will MAP_FIXED_NOREPLACE then work in this case?

Does it make a difference whether you use shm_open() or open() ?

I've looked into the code for boost interprocess and it does not seem to use either MAP_FIXED or sendmsg. It seems this is not supported.

The approach I am currently trying is simplifying:

process A:

fd = open("foobar", O_RDWR);
void* address = mmap(nullptr, PROT_READ|PROT_WRITE, MAP_SHARED, fd);
(*address) = address;

process B:

fd = open("foobar", O_RDONLY);
void* addressRequested = readAddressFromFoobar();       
void* address = mmap(addressRequested, PROT_READ, MAP_SHARED|MAP_FIXED, fd);

This fails with E_NOMEM. While if I replace addressRequested with a nullptr it succeeded and I can access the same data but can't rely on any pointers.

Can somebody demonstrate or link to a way to do this that works or explain definitively why this cannot ever work in current Linux.

I am quite aware that we can share objects using internal offsets instead of pointers but loses a lot of convenience. STL types generally assume they can use pointers. I do not wish to write my own versions of every container I wish to use if I can possibly avoid it.

---

Apparently boost works around this issue by providing containers that use smart pointers instead of raw pointers. I had not realised that. This is a good solution to the general problem but a different question from this one. Locating a canonical explanation of that would be useful as a better answer to Does boost interprocess support sharing objects containing pointers between processes? or indeed a better question.

Ronni

Say you did get both processes to agree on an virtual-addresses-range in advance; is there any guarantee that that range would still be unused/free-for-mapping a moment later on (when both processes actually try to map it to the shared memory region)? Seems iffy to me. – Sorption

One process would create the mmap without asking for a fixed address, the other processes would use the address given to that process. So the space is already allocated. – Ronni

Did you read the manual pages for the mmap system call? It answers your question. – Honeyman

Not in the second process; the second process's call to mmap() at that address could fail due to something else in the second process already using virtual memory addresses within that range, no? – Sorption

@SamVarshavchik Of course I read the man pages several times. It describes the function. It does not explain how to use it. The idea of using MAP_FIXED never occurred to me until I started trying to write this question. and found https://mcmap.net/q/874050/-how-do-i-choose-a-fixed-address-for-mmap as a result. The man page also warns that this is dangerous. I assumed MAP_FIXED was intended for device drivers rather than userspace applications. – Ronni

@JeremyFriesner I assume address space for each process is unique except for shared pages and that it is a finite resource allocated by the OS. Is this assumption incorrect? – Ronni

@BruceAdams Your assumption is correct. But as an example of what I meant, if your process B is using virtual-addresses 0x1000000 through 0x2000000 as part of its heap, and you try to mmap() your shared memory-region to start at virtual-address 0x1000005, then (hopefully) mmap() will fail because the virtual-addresses inside process B that it wants to redefine (to point to the shared-memory region) are already in use for another purpose. And with ASLR, there's (deliberately) no way to know what virtual-address ranges are going to be in use or not. – Sorption

Well, if everything in that manual page is read, it should be fairly clear that there are no guarantees, whatsoever, that even with MAP_FIXED the mapping will succeed. Address space layout randomization, a non-negotiable, mandatory, security requirement of all modern operating systems (including Linux) makes any kind of a fixed mapping a non-starter and completely impossible. The End. – Honeyman

@SamVarshavchik ASLR is not even mentioned in the man page. Though it is worth noting that what I get from "man mmap" on my system is not the same as what is written here which gives a little more information. My system is recent but only has v4.1.3 of the man pages whereas the link is to 5.13. – Ronni

@JeremyFriesner The v5.13 man page offers MAP_FIXED_NOREPLACE (added to kernel v4.17) which avoids collisions. The kernel on the main system I need to support is v4.18 (but has an out of date man page it seems). – Ronni

MAP_FIXED_NOREPLACE looks like it will keep the process from accidentally lobotomizing itself, but in the event of a would-be collision you're still left with the problem that your mmap() call has failed and so your process can't access the shared memory region. – Sorption

@JeremyFriesner As I said, I believe a collision is impossible if the result of a previous mmap not using MAP_FIXED is used to select the address. However, if you were doing something that did need MAP_FIXED. MAP_FIXED_NOREPLACE would let you replace an operations that will result in a segfault with some kind of negotiation. Tell the 'write' tochoose another location, rinse and repeat. I'm not sure why that would be necessary so it would be interesting to read up if there was a justification for adding NOREPLACE other than making the interface better. – Ronni

You can see code that runs on Linux and several Unix variants using mmap(MAP_FIXED) to implement a shared malloc pool at github.com/ChrisDodd/shm_malloc – Buttonhole

@ChrisDodd looks interesting. I can't quite work out how you guarantee the same fixed address is given to each process. Maybe with more reading. – Ronni

It just has a fixed base pointer in malloc.c which empirically works for each pointer size/platform of interest. Whether this is always safe, or only works when you're replacing malloc and no other shared libraries are calling mmap, I don't know. Anyway, my answer is just seeking to discover a suitable base at runtime, instead of hardcoding it and hoping it never changes. – Byerly

shm_open seems to be exactly what you are looking for. It maps the same memory pages on both process, but it also ensure that pages are coherent (that is, if a process writes to the memory page, and another process wants to read from that page, the cache will be flushed upon context switch so that the read works) – Wifely

Answer 1

Can somebody demonstrate or link to a way to do this that works or explain definitively why this cannot ever work in current Linux.

Your existing way can already be made to work.

It's a bit fragile, and fiddly, and I'm not sure it is a good idea, but it can be made to work.

1. ASLR

   Since two processes will (probably) have different ASLR offsets for the mmap base you can't just take the address returned by mmap in the first process and assume it will work in the second.

   With the current implementation, it should be sufficient to mmap a small area (with addr=NULL) to discover the randomized base address, and then request the real mapping at the next 1MB boundary.

2. Runtime variance

   In isolation this might be enough, but it's possible the other process has already consumed the same address. For example, a shared library could call mmap before the user code is reached, and this could vary on each run due to ASLR, library updates, preloads or something else.

   Probably the second process should also do a non-fixed mapping with no requested address to establish its own effective base (including both ASLR and any existing mmaps), so it can sanity-check the advertised address before trying it with MMAP_FIXED.

   So, you need a feedback/consensus mechanism. If the second process can't map at the first advertised address, have it create its own mapping and advertise that address back to the first process (which should release its own maps before trying, in case they overlap).

   This iteration also makes the first point less brittle in the face of future ASLR changes.

Note that even when this works, the fact that you don't differentiate between shared-address-space and private-address-space pointers is a significant risk. You can quite easily store a pointer to non-shared memory in one of your shared structures because they're the same type.

I'd strongly advise using the Boost.Interprocess allocators, offset_ptrs etc, to make the distinction explicit.

Byerly

Why do you need to " then request the real mapping at the next 1MB boundary." ? I think I may be misunderstanding how ASLR works. I assumed that the OS tries to avoiding giving processes address spaces that overlap. Is there in fact no such requirement? – Ronni

If you use offset_ptr then you don't need the base address to be the same anyway. For this case we can assume the shared pointers stay constrained to the type as they are provided by SharedAllocator. Caveat programmer as always. You would never let such a structure adopt a pointer allocated via a different allocator including a different instance of SharedAllocator. – Ronni

The OS doesn't need to care whether two process's virtual address spaces overlap, because they have independent page tables. There's definitely no requirement for processes to have non-overlapping address spaces. – Byerly

I imagine they are only partially independent. Many address translations are shared between processes as in for example https://mcmap.net/q/911851/-who-performs-the-tlb-shootdown . I would expect process to have unique mappings for their own code & data segments but shared ones for shared code and data including CoW children and shared memory. Put another way mmap could deliberately share the mapping for a deliberately shared block of memory if it doesn't overlap and you ask for it with MAP_FIXED_NOREPLACE. It is simply safer and easier to implement adding an entry to just the asking processes page table – Ronni

It seems a lot of my most recent questioning was prompted by a silly bug in some of my test code. MAP_FIXED didn't work for me when using a forked child process even though asking mmap() to return any address actually seems to reliably give the same address to the child. It actually works just fine for unrelated processes (bar the caveat of MAP_FIXED_NOREPLACE failing if the address is already in use). – Ronni

I tested an implementation that tries to use the same address for the second process. It sometimes works and sometimes fails. If I look at /proc/<pid>/maps for a failure the address is actually free in the second process. The rules on when it will or won't work are unclear. – Ronni

Can you explain the bit about "requesting the real mapping at the next 1MB boundary." It is not clear to me. – Ronni

ASLR (per the documentation) randomizes the base address within a fixed 1MB region. Rounding up to the next multiple of 1MB should give a correctly-aligned address which is the same in both processes even with (current) ASLR. ((uintptr_t >> 20) + 1) << 20 should work (or you can use bitmasks instead). It might be sensible to choose a 2MB or 4MB boundary instead if you have hugepages, but the idea is the same. – Byerly

It is not immediately obvious to me how mmap(nullptr,...) lets you discover the the base address and why that matters. So it is one of 256 possible pages in a 1MB region. How does that help us? Is it just in case the first process returns an area before the start of the second processes mmap area? – Ronni

We're trying to find an address that both processes will allow to be mapped. Rounding up to the next 1MB boundary may not be sufficient, but it seems like a reasonable starting point for step 2. – Byerly

Do you know what happens if you mmap the same shared memory region multiple times in the same process? Do you get the same address each time, a different address or is it undefined? I experimented with looping trying different addresses and when it fails no attempt a retrying with different addresses ever seems to succeed. Probably the best thing to do is parse /proc/self/maps. I think that kind of detail is needed to make this a +100 answer. – Ronni

You're just saying the feedback/consensus mechanism should be based on parsing /proc/self/mmaps instead of on calling mmap. That's fine, but the consensus mechanism itself is the hard part. – Byerly