Fork-join model implementation for Python? (Equivalent to Java's ForkJoinPool)

Question

I am looking for an implementation of the fork-join model for Python. As Java's ForkJoinPool, it should allow to split (fork) the work of a task into several sub tasks recursively. Once the sub tasks are completed, the results are joined and returned. Ideally, it should support threads and processes similar to the ThreadPoolExecutor and ProcessPoolExecutor in concurrent.futures, but threads are more important for now. It must allow to limit the number of threads (I want to have one thread per core). I am aware that this will only be useful if the code releases the GIL.

Example from Wikipedia to clarify fork-join model:

solve(problem):
    if problem is small enough:
        solve problem directly (sequential algorithm)
    else:
        for part in subdivide(problem)
            fork subtask to solve(part)
        join all subtasks spawned in previous loop
        return combined results

Is there such a library in Python? I could not find one.

Spectroscopy

You can try: docs.python.org/3/library/multiprocessing.html – Deuteranope

Yes, but then I have to implement the fork-join model on my own. – Spectroscopy

I think vorpus.org/blog/… reinvented fork-join in Trio library (they called it "nurseries") – Sis

Answer 1

I figured that what you want is to collect the result, multiprocessing.starmap() might be the choice, here goes the example

import multiprocessing as mp

def func(x, y):
    return x + y

l = list()
with mp.Pool(mp.cpu_count()) as p:
    l = p.starmap(func, [(1,2), (2,3), (3,4)])

print(l)  # result in [3, 5, 7]

Adobe

Sorry, I forgot to mention that the fork-join model works recursively. Now, you might say that then I can just call starmap again inside the function. Unfortunately, this does not work because if the recursion is deeper than the number of threads available, it will just block. – Spectroscopy