Create MPI processes on the fly with fork?

Question

If I use MPI, I have a number of processes specified when I run the main program. However I would like to start with one process and dynamically decide at runtime if and when I need more, to fork more processes off. Is that or something similar possible?

Otherwise I would have to reinvent MPI which I would very much like to avoid.

Canaveral

I refer you to my answer to [this question][1]. [1]: #9683831 – Schoolmate

@HighPerformanceMark: But that just starts a new process. This is not exactly a copy on write fork from the precise location (and configuration) of the current process. – Canaveral

you asked 'is that or something similar possible ?', I propose that mph_comm_spawn is similar. But if it isn't suitable for your requirements you may be left with having to rewrite MPI. Or use another parallelisation library / toolset / approach. – Schoolmate

Since you are speaking of forking, it looks like all your processes are running on the same node. MPI doesn't make this kind of assumption, and MPI processes might be scattered among many nodes on a cluster. This is why, as HighPerformanceMark says, the closest MPI operation to what you desire is a spawn. To do a kind of fork the MPI way, you'd have to spawn a new process and send it its initial state using P2P communications. – Luigi

Answer 1

It is not possible to use fork() as the child process will not be able to use MPI functions. There is a simple mechanism in MPI to create dynamically new processes. You must use the MPI_Comm_spawn function or the MPI_Comm_spawn_mutliple

OpenMPI doc: http://www.open-mpi.org/doc/v1.4/man3/MPI_Comm_spawn.3.php

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>

#define NUM_SPAWNS 2

int main( int argc, char *argv[] )
{
  int np = NUM_SPAWNS;
  int errcodes[NUM_SPAWNS];
  MPI_Comm parentcomm, intercomm;

  MPI_Init( &argc, &argv );
  MPI_Comm_get_parent( &parentcomm );
  if (parentcomm == MPI_COMM_NULL) {
    MPI_Comm_spawn( "spawn_example", MPI_ARGV_NULL, np, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, errcodes );
    printf("I'm the parent.\n");
  } else {
    printf("I'm the spawned.\n");
  }
  fflush(stdout);
  MPI_Finalize();
  return 0;
}

Eu