Comparison of multi-threading models in Julia >=1.3 and Python 3.x

Question

I would like to understand, from the user point of view, the differences in multithreading programming models between Julia >= 1.3 and Python 3.

Is there one that is more efficient than the other (in the sense that rising the thread numbers reduces more the computational time) ? In which situations (e.g. one model may have an edge, but only on computational or memory intensive tasks) ?

Is one that is more practical/provide higher level functions than the other ?

Is one that is more flexible than the other (e.g. it can be applied to a wider set of cases) ?

Scullion

Answer 1

There are several differences between the languages with Julia providing many levels of functionality on this what you can find in Python. You have the following types of parallelism (I am discussing here the standard language features not functionality available via external libraries):

1. SIMD (signle-instruction-multiple-data) feature of CPUs

• Julia: combine @simd with @inbounds (see https://docs.julialang.org/en/v1/manual/performance-tips/)
• Python: not supported

2. Green threads (also called Coroutines). (This is not an actual threading - but allows to use one system thread across many tasks. This is particularly useful to parallelize IO operations such as web scraping or inter-process communication - for an example if one task is waiting for IO, another tasks can execute in parallel.)

• Julia: use a combination of @sync (to collect a group of tasks) and @async (to spawn new tasks) macros (for more details see https://docs.julialang.org/en/v1/manual/parallel-computing/)
• Python: use asyncio (for more details see https://docs.python.org/3/library/asyncio-task.html)

3. Multihreading: run several tasks in parallel within a single process (and shared memory) across several system threads:

• Julia: use Threads.@threads macro to parallelize loops and Threads.@spawn to launch tasks on separate system threads. Use locks or atomic values to control the parallel execution. (for more details see https://docs.julialang.org/en/v1/manual/parallel-computing/)

• Python: not useful for CPU-dominated tasks due to GIL (global-interpreter-lock) (see the comment by @Jim below)

4. Multi-processing

• Julia: use macros from the Distibuted package to parallelize loops and spawn remote processes (for more details see https://docs.julialang.org/en/v1/manual/parallel-computing/)

• Python: use multiprocessing library - for more details see https://docs.python.org/3.8/library/multiprocessing.html

Chilung

Instead of "Python: not supported due to GIL," it would be more correct to say "Python: not useful for CPU-dominated tasks due to GIL." You can create single process, shared memory threads in Python (through its threading library, which largely has the same interface as its multiprocessing library), and you might want to do so for I/O bound tasks. And then another caveat: those threads can do useful CPU bound work if they primarily call out to an external library, like NumPy, that releases the GIL. (Not all external libraries release the GIL, but many do.) – Julieannjulien

Thanks for the nice comment! Isn't "not useful for CPU-dominated tasks" in practice the same as "use co-routines" which exactly can be used in I/O blocking scenarios. Regarding the caveat, out of curiosity, what happens when I have a numpy vector and 2 threads mutating it in the same time with a numpy method such as .add()? Is there a nice thread locking mechanism? – Chilung

If two threads mutate the same NumPy array in-place, the result will be a hardware race condition/undefined behavior, just as though you did it in C. If they had been Python lists, the GIL would have protected concurrent access, with a big performance cost. – Julieannjulien

Although I/O bound tasks may be implemented with async or threads, switching between them would be a big refactoring and someone might have reasons to prefer one over the other. So my comment was mostly for completeness. – Julieannjulien

I've been doing a deep-dive into Julia recently, and I'm astonished that the only way to set the number of physical (not green) threads is with an environment variable before Julia starts. Python lets you create a threading.Thread or a ThreadPoolExecutor, which usually gets gummed up because of the GIL, and Julia lets you run code truly in parallel, but you can't even make one new thread? (I've found @threadcall for external calls, though that's limited to a preset pool, too; by default, 4.) Not a complaint; just surprisingly hard-coded! – Julieannjulien

In the command line you have also the option ` -t, --threads {N|auto}` I do not remember at the moment whether it was added in Julia 1.6.0 or 1.7.0 (which is rc1 as of today). Once the Julia process runs the number of threads is fixed. And yes this is different than other langs. – Chilung