Ordering Output in MPI

Question

in a simple MPI program I have used a column wise division of a large matrix. How can I order the output so that each matrix appears next to the other ordered ? I have tried this simple code the effect is quite different from the wanted:

for(int i=0;i<10;i++)
{
    for(int k=0;k<numprocs;k++)
    {
        if (my_id==k){
            for(int j=1;j<10;j++)
                printf("%d",data[i][j]);
        }
        MPI_Barrier(com);
    }
    if(my_id==0)
        printf("\n");
}

Seems that each process has his own stdout and so is impossible to have ordered lines output without sending all the data to one master which will print out. Is my guess true ? Or what I'm doing wrong ?

Gouveia

Note: I have also tried fflush the output of each process.... not obtaining the result I wanted ;-). – Gouveia

Answer 1

You guessed right. The MPI standard does not specify how stdout from different nodes should be collected for printing at the originating process. It is often the case that when multiple processes are doing prints the output will get merged in an unspecified way. fflush doesn't help.

If you want the output ordered in a certain way, the most portable method would be to send the data to the master process for printing.

For example, in pseudocode:

if (rank == 0) {
    print_col(0);
    for (i = 1; i < comm_size; i++) {
        MPI_Recv(buffer, .... i, ...);
        print_col(i);
    }
} else {
    MPI_Send(data, ..., 0, ...);
}

Another method which can sometimes work would be to use barries to lock step processes so that each process prints in turn. This of course depends on the MPI Implementation and how it handles stdout.

for(i = 0; i < comm_size; i++) {
    MPI_Barrier(MPI_COMM_WORLD);
    if (i == rank) {
         printf(...);
    }
}

Of course, in production code where the data is too large to print sensibly anyway, data is eventually combine by having each process writing to a separate file and merged separately, or using MPI I/O (defined in the MPI2 standards) to coordinate parallel writes.

Emetine

Omg thank you for this answer if you are still around. Had a somewhat different problem, but thought I was doing something else wrong to mess up the output. I'd seen something about having one thread handle the output but was struggling with how to do it properly. Probably saved me days of frustration after having days of it already. :) – Scruff

Answer 2

I produced ordered output to a file before using the exact same method. You could try printing to a temporary file, printing the contents of said file and then deleting it.

Flywheel

Of course this is not actually defined to work as such and should not be used in any final product. But, I too found that by redirecting the standard output of my mpirun command to a file, the output became ordered with respect to time. this was not the case when std output was directed to the terminal. Thanks for the idea. This turned out to be a quick and easy method to debug a race condition in my code. – Fez

This may be a result of collective buffering (tasks gather write data to one designated i/o task) when writing to files and could possibly be disabled in certain environments. – Rozella

Answer 3

Have the root processor do all of the printing. Use MPI_Send/MPI_Recv or MPI_Gather (or whatever) to send the data in turn from each processor to the root.

Ruddy

Answer 4

To solve this problem you can use short sleep. I use and then it works in 99%

printf("text nr 1\n");
MPI_Barrier(MPI_COMM_WORLD);
usleep(100);
printf("text nr 2\n");

It's not very elegant but works.

Clift