How do I run multiple subprocesses in parallel and wait for them to finish in Python

Question

I am trying to migrate a bash script to Python.

The bash script runs multiple OS commands in parallel and then waits for them to finish before resuming, ie:

command1 &

command2 &

.

commandn &

wait

command

I want to achieve the same using the Python subprocess. Is this possible? How can I wait for a subprocess? call command to finish before resuming.

Orometer

possible duplicate of Multiprocessing vs Threading Python – Fireguard

Answer 1

You can still use Popen which takes the same input parameters as subprocess.call but is more flexible.

subprocess.call: The full function signature is the same as that of the Popen constructor - this functions passes all supplied arguments directly through to that interface.

One difference is that subprocess.call blocks and waits for the subprocess to complete (it is built on top of Popen), whereas Popen doesn't block and consequently allows you to launch other processes in parallel.

Try the following:

from subprocess import Popen
commands = ['command1', 'command2']
procs = [ Popen(i) for i in commands ]
for p in procs:
   p.wait()

Arlberg

In your code, command1 and command2 get executed simultaneously? Or command2 gets initiated after command1 is done? – Hammonds

I've tested, they run in parallel. You can substitute .wait() with .communicate() and add arguments stdout=subprocess.PIPE and stderr=subprocess.PIPE to Popen() if you want to capture stdout/sterr of the child process instead of letting them clump together with the parent's stdout/stderr – Kudos

They do run in parallel since the subprocesses start when Popen returns. But they block at wait(). So what this code does is, loop once in the array of procs and blocks at the first wait() until it is over. Then blocks at the next wait() if the subprocess is still running. – Edomite

in case this helps anyone - the elements in procs execute upon declaration. so the list is functioning more as an action list than a declaration list. the second for loop goes so quickly that the wait action is initiated probably before the commands execute – Lessee

As a useful application, commands could be a list of python scripts. Use Popen(["python", i]) to execute them. Output could be directed to a log file using with open(log_file, "w+" as f: ... Popen(["python", i], stderr=f) – Obbligato

Answer 2

Expanding on Aaron and Martin's answer, here is a solution that runs uses subprocess and Popen to run n processes in parallel:

import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5'] 
n = 2 #the number of parallel processes you want
for j in range(max(int(len(commands)/n), 1)):
    procs = [subprocess.Popen(i, shell=True) for i in commands[j*n: min((j+1)*n, len(commands))] ]
    for p in procs:
        p.wait()

I find this to be useful when using a tool like multiprocessing could cause undesired behavior.

Manas

does not reach cmd5 – Lessee

@rictuar: yes, I posted a fix below – Godolphin

Answer 3

Noah Friedman's answer is nice but fails for some combinations of n and number of commands by skipping some final commands. I changed it to this, which i find much more readable:

import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5']
cpus = 2
while commands:
    batch = commands[:cpus]
    commands = commands[cpus:]
    procs = [subprocess.Popen(i, shell=True) for i in batch]
    for p in procs:
        p.wait()

Also note that this is not optimal, as a single hanging process holds back the next batch from being processed. The delay is actually noticeable. This is better:

import subprocess
from concurrent.futures import ProcessPoolExecutor

def my_parallel_command(command):
    subprocess.run(command, shell=True)

commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5']
cpus = 2
with ProcessPoolExecutor(max_workers = cpus) as executor:
    futures = executor.map(my_parallel_command, commands)

Harte

that was nice of you – Lessee