Get return value for multi-processing functions in python

Question

I have two functions to run in parallel and each of them returns a value. I need to wait for both functions to finish and then process the returns from them. How could I achieve this in python. Assume

def fun1():
   #do some calculation#
   return val1

def fun2():
   #do some calculation#
   return val2

I want fun1 and fun2 to run in parallel and then need to calculate

valsum = val1+val2

Upsurge

Answer 1

Using concurrent.futures:

from concurrent.futures import ProcessPoolExecutor as Executor
#from concurrent.futures import ThreadPoolExecutor as Executor  # to use threads
with Executor() as executor:
    future1 = executor.submit(fun1, arg1, arg2, ...)
    future2 = executor.submit(fun2, arg1, arg2, ...)
    val1, val2 = future1.result(), future2.result()
    valsum = val1 + val2

concurrent.futures.Executor.submit schedules the function to be executed, and returns a concurrent.futures.Future object.

concurrent.futures.Future.resultreturns the return value returned by the function.

---

Using multiprocessing.pool:

from multiprocessing.pool import Pool
#from multiprocessing.pool import ThreadPool as Pool  # to use threads
with multiprocessing.pool.Pool() as pool:
    result1 = pool.apply_async(fun1, (arg1, arg2, ...))
    result2 = pool.apply_async(fun2, (arg1, arg2, ...))
    val1, val2 = result1.get(), result2.get()
    valsum = val1 + val2

multiprocessing.pool.Pool.apply_async returns an AsyncResult object, and AsyncResult.get will return the return value of the function once it finish and the result arrive.

Exigent

Using concurrent.futures works fine but not when using multiprocessing.pool. It gives an error that says AssertionError: daemonic processes are not allowed to have children when using the latter method. – Upsurge

@TheStupidOne, It works fine for me. Could you show the full traceback? – Exigent

The cleanest answer that I've seen in many places Most of the examples work with .map(), but that is not the only solution. – Selfsupporting

Answer 2

This is a simple example using list comprehension and calling .result() on each future which blocks until the process returns:

import concurrent.futures

def download_video(data):
    # print(data) 
    return data

with concurrent.futures.ProcessPoolExecutor() as executor:
    futures = [executor.submit(download_video, x, ) for x in your_list]
    results = [result.result() for result in futures]

Mortgagor