Shared memory in mpi4py

Question

I use a MPI (mpi4py) script (on a single node), which works with a very large object. In order to let all processes have access to the object, I distribute it through comm.bcast(). This copies the object to all processes and consumes a lot of memory, especially during the copying process. Therefore, I would like to share something like a pointer instead of the object itself. I found some features in memoryview useful to boost work with the object inside a process. Also the object's real memory address is accessible through the memoryview object string representation and can be distributed like this:

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

if rank:
    content_pointer = comm.bcast(root = 0)
    print(rank, content_pointer)
else:
    content = ''.join(['a' for i in range(100000000)]).encode()
    mv = memoryview(content)
    print(mv)
    comm.bcast(str(mv).split()[-1][: -1], root = 0)

This prints:

<memory at 0x7f362a405048>
1 0x7f362a405048
2 0x7f362a405048
...

That's why I believe that there must be a way to reconstitute the object in another process. However, I cannot find a clue in the documentation about how to do it.

In short, my question is: Is it possible to share an object between processes on the same node in mpi4py?

Billet

could you accept the new MPI 3.0 answer of JobJob as correct? – Lex

Answer 1

I don't really know much about mpi4py, but this should not be possible from the MPI point of view. MPI stands for Message Passing Interface, which means exactly that: pass messages around between processes. You could try and use MPI One-sided communication to resemble something like a globally accessible memory, but otherwise process memory is unavailable to other processes.

If you need to rely on a large block of shared Memory, you need to utilize something like OpenMP or threads, which you absolutely could use on a single node. A hybrid parallelization with MPI and some shared memory parallelization would allow you to have one shared memory block per node, but still the option to utilize many nodes.

Whitener

This answer is incorrect in light of MPI 3.0 shared memory (software.intel.com/en-us/articles/…) – Hexahydrate

@BenThompson OK, you want me to delete it or change the answer? I have 0 experience with this new feature... – Whitener

Answer 2

Here's a simple example of shared memory using MPI, very slightly modified from https://groups.google.com/d/msg/mpi4py/Fme1n9niNwQ/lk3VJ54WAQAJ

You can run it with: mpirun -n 2 python3 shared_memory_test.py (assuming you saved it as shared_memory_test.py)

from mpi4py import MPI 
import numpy as np 

comm = MPI.COMM_WORLD 

# create a shared array of size 1000 elements of type double
size = 1000 
itemsize = MPI.DOUBLE.Get_size() 
if comm.Get_rank() == 0: 
    nbytes = size * itemsize 
else: 
    nbytes = 0

# on rank 0, create the shared block
# on rank 1 get a handle to it (known as a window in MPI speak)
win = MPI.Win.Allocate_shared(nbytes, itemsize, comm=comm) 

# create a numpy array whose data points to the shared mem
buf, itemsize = win.Shared_query(0) 
assert itemsize == MPI.DOUBLE.Get_size() 
ary = np.ndarray(buffer=buf, dtype='d', shape=(size,)) 

# in process rank 1:
# write the numbers 0.0,1.0,..,4.0 to the first 5 elements of the array
if comm.rank == 1: 
    ary[:5] = np.arange(5)

# wait in process rank 0 until process 1 has written to the array
comm.Barrier() 

# check that the array is actually shared and process 0 can see
# the changes made in the array by process 1
if comm.rank == 0: 
    print(ary[:10])

Should output this (printed from process rank 0):

[0. 1. 2. 3. 4. 0. 0. 0. 0. 0.]

Manchineel

What about a multi-dimensional array, let's say np.zeros((50, 50, 50))? – Afb

@Zézouille for a multi-dim array it's similar, you'd just make (approximately) the following changes: size = 1000 to shape = (50,50,50), size = np.prod(shape) and this line becomes ary = np.ndarray(buffer=buf, dtype='d', shape=shape) # instead of shape=(size,) and to get this particular example to run you'd also want to change ary[:5] = np.arange(5) to ary[:5, 1, 1] = np.arange(5) and print(ary[:10]) to print(ary[:10, 1, 1]) – Manchineel

@Manchineel Do you have any idea if that would be possible with a dictionary? – Band