Python Multiprocessing Script Freezes Seemingly Without Error

Question

I am trying to use the multiprocessing package to call a function (let's call it myfunc) in parallel, specifically using pool.map i.e. pool.map(myfunc, myarglist). When I simply loop over myarglist without using multiprocessing there are no errors, which should be the case because all operations in myfunc are called within a try block. However, when I call the function using pool.map the script invariably stops running, i.e. it stop printing a "myfunc done!" statement within my function and the processes stop using the CPUs, but it never returns resultlist. I am running python 2.7 from the terminal in ubuntu 12.04. What could cause this to occur and how should I fix/troubleshoot the problem?

cpu_count = int(multiprocessing.cpu_count())
pool = Pool(processes = cpu_count)
resultlist = pool.map(myfunc, myarglist)
pool.close()

Update One issue when using multiprocessing can be the size of the object, if you think that may be a problem see this answer. As the answer notes "If this [solution] doesn't work, maybe the stuff you're returning from your functions is not pickleable, and therefore unable to make it through the Queues properly." Multiprocessing passes objects between processes by pickling them. It turns out that one or two of my objects had soup from BeautifulSoup that would not pickle.

Polyphagia

Impossible to know what's causing this without some information on the body of myfunc. – Boykin

Can you tell if the function has run at all? – Demmer

Yes, I know the the function runs because it prints out "myfunc done!" over and over. I tried removing the return statement from myfunc and the pool.map() call completes. So it seems to be having troubles at the stage where pool.map combines the results. In looking at the system monitor though nothing is happening in any of the python processes at this point. And it is taking longer than it would to simply run on a single core. – Polyphagia

removing the return makes the problem go away - is myarglist large or are you returning a particularly large object or one that has references to other objects? You may want to try map_async and use the callback to track how much gets done before all goes wrong. – Peshitta

So I have isolated an object that will not return when myfunc runs in parallel, but has no problem when myfunc is called directly. The object in question is about five times larger than the typical object in the list, but is otherwise normal. When I use map_async the function executes fine, but result.get() hangs. Interestingly(?), if and only if I call myfunc on that object not in parallel right before I call myfunc on the object using map_async I get the error message "Segmentation fault (core dumped)" before I even call result.get(). – Polyphagia

myfunc is a lot of code and calls several other functions. I am not going to post it here, but everything is done within a try statement and function itself does not seem to be the problem, but returning the function's result in parallel is. – Polyphagia

@Polyphagia Can you say more about what the large object actually looks like? I'm assuming it's a custom class of some kind? What's in it? What makes it 5x larger than the other objects you're working with? Based on your comment about the segfault, I'm wondering if your child process is segfaulting at some point while executing myfunc or trying to return from myfunc. If that is indeed happening, map would hang forever. – Souther

@Polyphagia Also, is the object you're talking about the object being returned from myfunc, or the object being passed to myfunc? – Souther

Answer 1

Check whether all the processes are started or not.This will help you to debug it.Also add Pool.join() at the end of your code.

This is a sample code

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':

    pool_size =2
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                )

    pool_outputs = pool.map(function_name,argument_list)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks

Volatilize