Program hangs in debug when multiprocessing process opens another process

Question

In a python program, a Process is opened using multiprocessing.Process. Then this process is creating a Pool in order to give it some work using the map() method.

When the program is normally run, all works as expected. However, when it is run in PyCharm debugger, the call to Pool.map never returns and program is locked.

The problem is demonstrated in the following simple example:

1) Code:

import multiprocessing

def inc(a):
    return a + 1;

def func():
    p = multiprocessing.Pool(2)
    print("before map")
    res = p.map(inc, [1,4])  # ==> the method hangs in debug.
    print("after call map")
    p.close()
    p.join()
    print(res)

def main():
    p = multiprocessing.Process(target=func)
    p.start()
    p.join()

if __name__ == '__main__':
    main()

2) Output as expected when program is run:

before map
after call map
[2, 5]

Process finished with exit code 0

3) Output when program run in debugger - never completes:

pydev debugger: process 13792 is connecting

Connected to pydev debugger (build 173.4301.16)
before map

Is this just a very annoying debugging issue (maybe caused by debugger background threads?)? or is it a multiprocessing problem that might appear also in real run?

It should be mentioned that using only one of the subprocessing steps, meaning either just opening a Process(), or just using a pool.map(), causes no problems and could be debugged. The problem occurs only in the "nested" subrocessing, as described.

I am running PyCharm on a Windows 10 64 bit machine.

Allysonalma

Got a similar problem in pycharm 2020. Any solutions you found? – Oldham

Unfortunately not, @MichelePiccolini. But I did not look into the issue since. – Allysonalma

Answer 1

I just had similar problem and found out that I was having a break point placed inside the function that I was calling using the map() method. Removing this break point or ignoring break points resolved the problem for me. Hope that this helps.

Tripp

Answer 2

I don't know exactly the cause of this, but I can say for my m1 mac, the hanging was caused by not explicitly setting the start method:

multiprocessing.set_start_method('spawn')

Wiegand