Popen waiting for child process even when the immediate child has terminated

Question

I'm working with Python 2.7 on Windows 8/XP.

I have a program A that runs another program B using the following code:

p = Popen(["B"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
return

B runs a batch script C. C is a long running script and I want B to exit even though C has not finished. I have done it using the following code (in B):

p = Popen(["C"])
return

When I run B, it works as expected. When I run A however, I expected it to exit when B exits. But A waits until C exits even though B has already exitted. Any ideas on what's happening and what possible solutions could be?

Unfortunately, the obvious solution of changing A to look like B is not an option.

Here is a functional sample code to illustrate this issue: https://www.dropbox.com/s/cbplwjpmydogvu2/popen.zip?dl=1

The zip file consists of the following files with the following contents:

A.py

from subprocess import PIPE, Popen
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("\r\n\r\nA: I'll wait for B\r\n")

p = Popen(["C:\\Python27\\python.exe", "B.py"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()

log("A: Done.\r\n")
sys.exit(0)

B.py

from subprocess import Popen, PIPE
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("B: launching C\r\n")

p = Popen(["C.bat"])

log("B: Not waiting for C at all. bye!\r\n")
sys.exit(0)

C.bat

@echo off
echo C: Start long running task : %time% >>  "log.txt"
ping -n 10 127.0.0.1>nul
echo C: Stop long running task : %time% >>  "log.txt"

Any input is much appreciated.

Shearwater

If I get this right, you have program Am which runs program B which runs program C. Program A also runs program C. Is this correct? – Guayaquil

No, program A doesn't run program C directly. It would be great if you took a look at attached example. Thanks. – Shearwater

So the second Popen is from program B? – Guayaquil

Yes. A only runs B and has nothing to do with C directly. – Shearwater

Does supplying close_fds=True to the second Popen() call help? I'm guessing that C inherits the stdout/stderr pipes from A and thus A waits until C closes them. – Boot

I tried using close_fds=True in second program (B) and it doesn't seem to make a difference. Could you please check the attached sample and see if something can be done? – Shearwater

Answer 1

You could provide start_new_session analog for the C subprocess:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

[1]: Process Creation Flags for CreateProcess()

Magnetoelectricity

Successfully used this code to create the process group (but without the DETACHED_PROCESS flag) and psutil to kill the process tree (ie. group): #1231169 – Groark

This only works for me on windows if I had close_fds=True to the Windows kwargs line: kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True) (see this answer) – Genro

@Genro have you noticed that it says that you cannot use close_fds=True on Windows if you redirect any of stdin,stdout, stderr? – Magnetoelectricity

@J.F.Sebastian Yes, I interpreted the docs to mean that if you use close_fds=True, stdin, stdout, and stderr will not work (although the phrasing is a bit ambiguous as to what happens if you actually do use them). Your snippet works very well for me after simply adding close_fds, but it would probably be safer to omit stdin/stdout/stderr when the platform is Windows. – Genro

@jtpereyda: the intent of the code is std*=DEVNULL i.e., you can't omit stdin/stdout/stderr here (at least on POSIX -- close_fds=True does not close std* there). You don't need close_fds=True unless you have other (non-standard) file descriptors opened (either close them or pass close_fds=True -- default on Python 3 on POSIX). On Windows, close_fds=True is enough (without redirecting) so that the child process won't inherit parent's file descriptors (I haven't tested what happens if you write to stdout in this case or pass std*=DEVNULL and close_fds=True together on Windows). – Magnetoelectricity

Combining DETACHED_PROCESS and CREATE_NEW_PROCESS_GROUP makes no immediate sense. Every process is in a group in Windows, even if it's just the group for the winlogon.exe or wininit.exe session, but the only API that uses process groups is GenerateConsoleCtrlEvent for sending Ctrl+Break (or Ctrl+C if the process manually enables it, since creating a new group initially disables it) to a group of processes that are attached to one's current console. Injecting the control thread is coordinated between the console instance (conhost.exe) and the Windows session server (csrss.exe). – Thermopile

The equivalent of a Unix process tree in Windows is a Job object. The process should be started suspended (creation flag CREATE_SUSPENDED) to ensure it's added to the job before it can spawn other processes that leak out of the job. This works best in Windows 8+ since it supports nested jobs, whereas older versions only allow a single job per process. – Thermopile

Note that starting from Python 3.7, on Windows it's no longer necessary to set close_fds=True - that is now the default and works together with redirecting stdin/stdout/stderr. Details in bug tracker here: bugs.python.org/issue19764 – Peasant

Answer 2

An alternative would be to start C as fully forked-off process in a separate process tree, e.g. via the start command of cmd.exe:

import subprocess
subprocess.Popen(["cmd.exe", "/C", "start notepad"])

Obviously, since it is completely independent, you cannot communicate with it. But you can use psutil to retrieve it's PID to at least monitor or close it, if necessary:

import psutil
for process in psutil.process_iter():
    if process.name() == 'notepad.exe':
        print(process.pid)

Gallstone

Answer 3

Here is a code snippet adapted from Sebastian's answer and this answer:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

I've only tested it personally on Windows.

Genro

Note that starting from Python 3.7, on Windows it's no longer necessary to set close_fds=True - that is now the default and works together with redirecting stdin/stdout/stderr. Details in bug tracker here: bugs.python.org/issue19764 – Peasant