MPI gather/reduce operation confusion?

Question

In the boost tutorial there are examples of gather and reduce operations. The code for gather is as follows:

#include <boost/mpi.hpp>
#include <iostream>
#include <vector>
#include <cstdlib>
namespace mpi = boost::mpi;

int main(int argc, char* argv[])
{
  mpi::environment env(argc, argv);
  mpi::communicator world;

  std::srand(time(0) + world.rank());
  int my_number = std::rand();
  if (world.rank() == 0) {
    std::vector<int> all_numbers;
    gather(world, my_number, all_numbers, 0);
    for (int proc = 0; proc < world.size(); ++proc)
      std::cout << "Process #" << proc << " thought of "
                << all_numbers[proc] << std::endl;
  } else {
    gather(world, my_number, 0);
  }

  return 0;
}

And the example for reduce is as follows:

#include <boost/mpi.hpp>
#include <iostream>
#include <cstdlib>
namespace mpi = boost::mpi;

int main(int argc, char* argv[])
{
  mpi::environment env(argc, argv);
  mpi::communicator world;

  std::srand(time(0) + world.rank());
  int my_number = std::rand();

  if (world.rank() == 0) {
    int minimum;
    reduce(world, my_number, minimum, mpi::minimum<int>(), 0);
    std::cout << "The minimum value is " << minimum << std::endl;
  } else {
    reduce(world, my_number, mpi::minimum<int>(), 0);
  }

  return 0;
}

In both of these examples we have an if/else conditional as follows:

if(world.rank() == 0){
  //gather or reduce operation
  //Output
}else{
  //gather or reduce operation
}

What I fail to understand here is the difference between the collective operation in the if vs. whats in the else? There's a difference in the number of parameters but I don't quite understand how the logic works out.

Thanks

Din

Answer 1

MPI has two types of collective operations - ones that have a designated "root" process and others that have not. Operations with designated root are broadcast (root process sends the same data to all processes), scatters (root process scatters its data to all processes), gathers (root process collects data from all processes) and reductions (root process gathers data from all processes while applying reduction over it). In the MPI standard those operations typically have forms similar to:

MPI_SCATTER(sendbuf, sendcount, sendtype,
            recvbuf, recvcount, recvtype, root, comm)

This MPI call has both input and output arguments and it has to be used in all processes, including the root one, but the input arguments (sendbuf, sendcount, sendtype) are only significant in the process whose rank equals root and are ignored in all other processes.

MPI was designed for portability and thus MPI calls were designed so that they can be implemented in the same way in both C and Fortran 77 - both languages did not support function overloading or optional arguments at the time when the MPI standard was designed. (Un-)fortunately C++ libraries like boost::mpi have taken on the liberty that C++ provides in overloading functions by providing versions of those calls that effectively hide the unused arguments. Now it is immediately obvious that a call to gather(world, my_number, 0) has no output argument and hence it has to be used in processes that are not the root of the operation while gather(world, my_number, all_numbers, 0) has an output argument and hence has to be used only in the root. This creates some asymmetry in writing the code - you have to do things like if (world.rank() == root) { ... } else { ... }.

I as a hardcore MPI user consider this ugly but there are other people who do not share my opinion. I guess... it depends.

Joinder

The C++ wrapper is indeed useless: it is just syntactic sugar and prevents to access to implementor-specific extensions. – Stirrup

Answer 2

Have a look in:

boost/mpi/collectives/gather.hpp

You'll see different implementations of gather_impl, one with "out" values in the argument list and one without. The one without "out" values sends its data to the root process. So in the gather example:

gather(world, my_number, 0) 

implements an mpi send and

gather(world, my_number, all_numbers, 0) 

implements a local copy for the root processor and an mpi receive for other processors.

Deandreadeane