Can a spawned process communicate with the "main" MPI communicator

Question

Is there a way using MPI to let spawned processes communicate with all other actors in the MPI_WORLD and not only with the parent that spawned the process?

Now I have two main agents, the so-called master and slave that run the following code (spawn.py):

# Spawn test: master and first slave
import sys

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()

if rank == 0 : # master code
    print "i am the master on rank %i" % (rank)
    running = True
    while running :
        msg = comm.recv(source=MPI.ANY_SOURCE,tag=0)
        print "master received message: ", msg
        if msg == "Done" :
            running = False
    print "master is done"

if rank == 1 : # slave code
    no_spawn = 1
    print "I am a slave on rank %i, about the spawn lower slaves" % (rank)
    icomm = MPI.COMM_SELF.Spawn(sys.executable,args=["Cpi.py","ben"],maxprocs=no_spawn)
    comm.send("Test_comm",dest=0,tag=0)
    icomm.send("Test_icomm",dest=0,tag=0)
    isize =  icomm.Get_size()
    print "on slave, isize= %i" % (isize)
    rec = 0
    while rec <= (no_spawn-1) :
        msg = icomm.recv(source=MPI.ANY_SOURCE,tag=20)
        print "slave received message: %s (rec=%i)" % (msg, rec)
        rec = rec +1
    import time
    print "slave going to sleep\n"
    time.sleep(1)
    for i in range(no_spawn) :
        message = ("To spawn from slave",)
        icomm.send(message,dest=i,tag=0)
    for i in range(no_spawn) :
        message = ("Done",)
        icomm.send(message,dest=i,tag=0)

    msg = comm.recv(source=MPI.ANY_SOURCE,tag=0)
    print "slave received message: ", msg

    comm.send("Done",dest=0,tag=0)

    MPI.Finalize()

The slave, in turn, spawns 1 more processes that runs the following code (CPi.py, named after the mpi4py tutorial file):

#!/usr/bin/env python

import sys

from mpi4py import MPI
comm = MPI.COMM_WORLD
icomm = MPI.Comm.Get_parent()
irank = icomm.Get_rank()

print "Spawn irank=%i" % (irank)
message = "From_Spawn_%i"%(irank)
icomm.send(message,dest=0,tag=20)

running = True
while running :
    msg = icomm.recv(source=MPI.ANY_SOURCE,tag=0)
    print "Spawn on irank %i received message: %s " %(irank,msg)
    if msg[0] == "Done" :
        running = False    

print "spawn %i sending a last msg to the master and the slave" % (irank)
comm.send(("To master from spawn",), dest=0,tag=0)
comm.send(("To slave from spawn",), dest=0,tag=0)

Between the master and slave I can send messages by using the comm communicator. Between the slave and the spawned process I can send messages over the icomm communicator. But what I really want is to spawn a process and that this process can communicate with both the master and slave over the comm communicator; see the last two lines of the spawned process. Is that possible? And would it be possible for the spawned process to listen as well to the main comm used by the slave and the master? Which rank would it be send to / listen to?

The provided code does not terminate because the last two messages send by the spawned process are neither received by the slave or the master. (I run the code with mpiexec -n 2 python spawn.py)

Ink

Answer 1

For the process spawned by the slave to talk to the master, it would need to create another new communicator using something like MPI_CONNECT and MPI_ACCEPT. It's possible to do, but you'd have to use the slave to transfer the connection details between the two.

Before you go through all of that, be sure that you can't just start your job with more process and arbitrarily assign different roles to different ranks. It's a pain to use intercommunicators under the best of circumstances and it's probably simpler to start out with the correct number of processes.

Uneducated

The merge commands from you other post wouldn't do? I could rewrite the code and initialize 1 master, n slaves and n spawnees, but for efficiency it seems important that the spawnees and slaves are on the same processors. I tried 1. function calls 2. threads, 3. MPI.Spawn. With 1 & 2 I cannot dynamically arrange communication between slave and spawnees. With 3. I can communicate between slave and spawnee but not communicate between slaves concurrently: either icomm.recv or comm.recv – Ink

You're correct that you can't do this with merge. It doesn't merge two communicators, just an intercommunicator. – Uneducated