Parrallel queue system, is there a non-blocking equivalent to gather? mpi4py

I would like to create a loop over an list of parameters (say 50). The loop should be shared by several processors (say 4) in parallel. I would like each processor to pick the next parameter from the list and call a function with this parameter. The function evaluations are expected to take unequal amounts of time. Here is a sketch of how I would like the jobs to be scheduled:

pr0 ===job0=== | ========job5======= | 
pr1 =====job1======= | ==job7== |
pr2 ===job2=== | ===job6=== | 
pr3 =job3= | =========job4======== |

I would like each process to be aware of the ids of the currently running jobs, so it can pick the next element in the list and start working with it. What I have tried so far is using comm.allgather():

from mpi4py import MPI
from random import random
from time import sleep


##### function that just wastes 0-10sec of our time
def run_test(counter, param ):
    time_to_waste = random() * 10
    sleep( time_to_waste )

    print " run job nr: ", counter, "with parameters: ", param, "in :", time_to_waste, "s"
    return 


#############   MAIN 
if __name__ == "__main__":
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()
    size = comm.Get_size()

    parameters = [random()] * 50
    counter = rank 

    while counter <= 50:

        param = parameters[counter]
        run_test( counter, param ) 

        print rank, comm.allgather(counter)
        counter = max( comm.allgather(counter) ) + size

        counter += size

However this creates a barrier, where processes are waiting for each other to finish their respective jobs. Of course this is totally inefficient, it would be better to split the array between the processors from the beginning. My jobs are being run like this:

pr0 ===job0===       | ========job4=======   | 
pr1 =====job1======= | ==job5==              |
pr2 ===job2===       | ===job6===            | 
pr3 =job3=           | =========job7======== |

In other words I would like a process to be able to access the counter variable from all other processes at all times. Can I make a non-blocking send from the processes before they start a job, and make their counters available for the other processes. Maybe sharing the memory for this variable is another way to go about it.

Is there a way to achieve this without dedicating a process to do the counting?

Recommended topics

Hot tags