Python multiprocessing example not working

Question

I am trying to learn how to use multiprocessingbut I can't get it to work. Here is the code right out of the documentation

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

it should output

>>> 'hello bob'

but instead i get

>>>

no errors or other messages, it just sits there, It is running in IDLE from a saved .py file on a Windows 7 machine with the 32-bit version of Python 2.7

Natashianatassia

Is your print statement really not intended under the def? – Raynaraynah

@Green Do you mean indented? Yes it is properly indented in the actual code, that was an error I made in posting – Natashianatassia

did you save the code in a file and run it? – Lamee

@ReutSharabani Yes I did, does that matter? – Natashianatassia

@ChuckFulminata Just making sure you're not doing it from the REPL thing. – Lamee

@ReutSharabani REPL? I am not familiar with that term – Natashianatassia

Python comes with an interperter (I think it's a [R]ead [E]val [P]rint [L]oop) and you can run stuff there. This is not the way you should run this example. You've written you're running it from a .py file on your HD. How are you executing that file? – Lamee

@ChuckFulminata The problem is trying to print to stdout when there is no stdout. The solution is to start IDLE (or any gui editor) in Command Prompt. See my answer. – Serrano

Answer 1

My guess is that you are using IDLE to try to run this script. Unfortunately, this example will not run correctly in IDLE. Note the comment at the beginning of the docs:

Note Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter.

The __main__ module is not importable by children in IDLE, even if you run the script as a file with IDLE (which is commonly done with F5).

Cadmarr

I'm not running it from the interactive interpreter – Natashianatassia

@Chuck Fulminata Where are you running it from? – Cadmarr

a .py file saved on the hard drive – Natashianatassia

@Chuck Fulminata And how are you running that file? – Cadmarr

the exact procedure I go through to run it is right click, edit with IDLE, run module(f5) – Natashianatassia

@Chuck Fulminata That is precisely the issue. That won't work, since the __main__ module is not importable by children in IDLE, EVEN IF you are running the script as a .py file. – Cadmarr

Oh, I didn't realize that, is that noted somewhere that I missed? – Natashianatassia

let us continue this discussion in chat – Natashianatassia

@JustinBarber Your repeated claim that the user code in a .py file cannot be imported is wrong. The problem is trying to 'print' in a process without stdout. The solution is to make sure there is an stdout. See my answer. – Serrano

@πόδαςὠκύς Can you please help me with this, I do not know what to do, and it seams like similar problem: #59892969 – Sweepback

Answer 2

The problem is not IDLE. The problem is trying to print to sys.stdout in a process that has no sys.stdout. That is why Spyder has the same problem. Any GUI program on Windows is likely to have the same problem.

On Windows, at least, GUI programs are usually run in a process without stdin, stdout, or stderr streams. Windows expects GUI programs to interact with users through widgets that paint pixels on the screen (the G in Graphical) and receive key and mouse events from Windows event system. That is what the IDLE GUI does, using the tkinter wrapper of the tcl tk GUI framework.

When IDLE runs user code in a subprocess, idlelib.run runs first, and it replaces None for the standard streams with objects that interact with IDLE itself through a socket. Then it exec()s user code. When the user code runs multiprocessing, multiprocessing starts further processes that have no std streams, but never get them.

The solution is to start IDLE in a console: python -m idlelib.idle (the .idle is not needed on 3.x). Processes started in a console get std streams connect to the console. So do further subprocesses. The real stdout (as opposed to the sys.stdout) of all the processes is the console. If one runs the third example in the doc,

from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

if __name__ == '__main__':
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

then the 'main line' block goes to the IDLE shell and the 'function f' block goes to the console.

This result shows that Justin Barber's claim that the user file run by IDLE cannot be imported into processes started by multiprocessing is not correct.

EDIT: Python saves the original stdout of a process in sys.__stdout__. Here is the result in IDLE's shell when IDLE is started normally on Windows, as a pure GUI process.

>>> sys.__stdout__
>>> 

Here is the result when IDLE is started from CommandPrompt.

>>> import sys
>>> sys.__stdout__
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
>>> sys.__stdout__.fileno()
1

The standard file numbers for stdin, stdout, and stderr are 0, 1, 2. Run a file with

from multiprocessing import Process
import sys

def f(name):
    print('hello', name)
    print(sys.__stdout__)
    print(sys.__stdout__.fileno())
if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

in IDLE started in the console and the output is the same.

Serrano

Answer 3

It works.

I've marked the changes needed to make your sample run using comments:

from multiprocessing import Process

def f(name):
print 'hello', name #indent

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()` # remove ` (grave accent)

result:

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

Output from my laptop after saving it as ex1.py:

reuts@reuts-K53SD:~/python_examples$ cat ex1.py 
#!/usr/bin/env python
from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
reuts@reuts-K53SD:~/python_examples$ python ex1.py 
hello bob

Lamee

Both of those were errors I made when I posted the question, neither are present in the actual code that won't do anything – Natashianatassia

As you can see, this code does work. Either you're not using the same code I've posted, or the problem isn't the code. Where are you running the code? How are you running it? More information is needed. – Lamee

The problem may be specific to Windows, I don't know. Even on Windows, if you run IDLE from a command line, the output is visible. See my answer. – Serrano

Answer 4

I had the issue that multiprocessing did not work on Spyder, and always landed here. I solved it by using threading instead of multiprocessing. as described here: https://pymotw.com/2/threading/

import threading
def worker(num):
    """thread worker function"""
    print 'Worker: %s' % num
    return
threads = []
for i in range(5):
    t = threading.Thread(target=worker, args=(i,))
    threads.append(t)
    t.start()

Tetrahedral

Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – Subjectify

Sami, if you can start Spyder from a console command line, the output should be visible, as it is for IDLE. (Are you running on Windows or ???.) – Serrano

Terry, I have not tried it and yes I'm running it on Windows. Karl: thank you I have added the examples. – Tetrahedral

Answer 5

Most likely your main process exits before sysout is flushed. Try this:

from multiprocessing import Process
import sys

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
    # make sure all output has been processed before we exit
    sys.stdout.flush()

If this doesn't work, try adding time.sleep(1) as the last statement.

Scevor

Though that might work, remember this code is verbatim from the documentation, and another poster has already noted that it works – Natashianatassia

The code is fine, you just can't see the output. Multiprocessing behaviour depends on the particular machine you run it on, and yours might just be fast enough to swallow the output before you can see it. – Scevor

Downvote all you like but please leave a comment so I can improve the answer. – Scevor

Flushing is not the issue when there is no stdout to flush. Having a stdout stream solve the problem. See my answer. – Serrano

Answer 6

Try using this code (from standard manual). Works for me on windows. Another one did not work for me either :)

import multiprocessing as mp

    def foo(q):
        q.put('hello')
    
    if __name__ == '__main__':
        mp.set_start_method('spawn')
        q = mp.Queue()
        p = mp.Process(target=foo, args=(q,))
        p.start()
        print(q.get())
        p.join()

Johannajohannah