subprocess: deleting child processes in Windows

Question

On Windows, subprocess.Popen.terminate calls win32's TerminalProcess. However, the behavior I see is that child processes of the process I am trying to terminate are still running. Why is that? How do I ensure all child processes started by the process are killed?

Twocycle

Here are 2 options 1. Use this exe as a subprocess which kills process trees for you: latenighthacking.com/projects/2002/kill 2. Convert the following C code into Python with ctypes: #1173842 – Outfielder

Which version of Python and Windows? – Fanniefannin

Answer 1

By using psutil:

import psutil, os

def kill_proc_tree(pid, including_parent=True):    
    parent = psutil.Process(pid)
    children = parent.children(recursive=True)
    for child in children:
        child.kill()
    gone, still_alive = psutil.wait_procs(children, timeout=5)
    if including_parent:
        parent.kill()
        parent.wait(5)

me = os.getpid()
kill_proc_tree(me)

Bystander

That code looks like it will only kill first-level children, not grandchildren etc. Might be an issue if you're launching build tools or .bat/.cmd files with cmd.exe. Unless get_children() means grandchildren too? – Warrick

Exactly. To include grandchildren you should specify the 'recursive' option as in parent.get_children(recursive=True) – Langley

This is an old answer, but it was exactly what I was looking for -- a cross-platform way of killing all child processes. All the other answers that showed up on Google claim it cannot be done. Thank you for psutil! – Dionnedionysia

Note that Windows doesn't maintain a process tree. parent.children(recursive=True) is building a tree on the fly by linking parent to child, so it won't find orphaned processes (i.e. if the parent died). – Etti

Thanks very much for this! It's exactly what I was looking for. Would you agree to let the code in this answer to be used under open source licensing terms? BSD or MIT would be ideal, since they are compatible with Numpy, Pandas, Scipy, etc. – Kenneth

It doesn't work all time. If process can spawn randomly. – Gingras

Answer 2

Use taskkill with the /T flag

p = subprocess.Popen(...)
<wait>
subprocess.call(['taskkill', '/F', '/T', '/PID', str(p.pid)])

The flags to taskkill has the following docs:

TASKKILL [/S system [/U username [/P [password]]]]
         { [/FI filter] [/PID processid | /IM imagename] } [/T] [/F]

/S    system           Specifies the remote system to connect to.
/U    [domain\]user    Specifies the user context under which the
                       command should execute.
/P    [password]       Specifies the password for the given user
                       context. Prompts for input if omitted.
/FI   filter           Applies a filter to select a set of tasks.
                       Allows "*" to be used. ex. imagename eq acme*
/PID  processid        Specifies the PID of the process to be terminated.
                       Use TaskList to get the PID.
/IM   imagename        Specifies the image name of the process
                       to be terminated. Wildcard '*' can be used
                       to specify all tasks or image names.
/T                     Terminates the specified process and any
                       child processes which were started by it.
/F                     Specifies to forcefully terminate the process(es).
/?                     Displays this help message.

Or walk the process tree using comtypes and win32api:

def killsubprocesses(parent_pid):
    '''kill parent and all subprocess using COM/WMI and the win32api'''

    log = logging.getLogger('killprocesses')

    try:
        import comtypes.client
    except ImportError:
        log.debug("comtypes not present, not killing subprocesses")
        return

    logging.getLogger('comtypes').setLevel(logging.INFO)

    log.debug('Querying process tree...')

    # get pid and subprocess pids for all alive processes
    WMI = comtypes.client.CoGetObject('winmgmts:')
    processes = WMI.InstancesOf('Win32_Process')
    subprocess_pids = {} # parent pid -> list of child pids

    for process in processes:
        pid = process.Properties_('ProcessID').Value
        parent = process.Properties_('ParentProcessId').Value
        log.trace("process %i's parent is: %s" % (pid, parent))
        subprocess_pids.setdefault(parent, []).append(pid)
        subprocess_pids.setdefault(pid, [])

    # find which we need to kill
    log.debug('Determining subprocesses for pid %i...' % parent_pid)

    processes_to_kill = []
    parent_processes = [parent_pid]
    while parent_processes:
        current_pid = parent_processes.pop()
        subps = subprocess_pids[current_pid]
        log.debug("process %i children are: %s" % (current_pid, subps))
        parent_processes.extend(subps)
        processes_to_kill.extend(subps)

    # kill the subprocess tree
    if processes_to_kill:
        log.info('Process pid %i spawned %i subprocesses, terminating them...' % 
            (parent_pid, len(processes_to_kill)))
    else:
        log.debug('Process pid %i had no subprocesses.' % parent_pid)

    import ctypes
    kernel32 = ctypes.windll.kernel32
    for pid in processes_to_kill:
        hProcess = kernel32.OpenProcess(PROCESS_TERMINATE, FALSE, pid)
        if not hProcess:
            log.warning('Unable to open process pid %i for termination' % pid)
        else:
            log.debug('Terminating pid %i' % pid)                        
            kernel32.TerminateProcess(hProcess, 3)
            kernel32.CloseHandle(hProcess)

Warrick

Answer 3

Here's example code for the Job object method, but instead of subprocess it uses win32api.CreateProcess

import win32process
import win32job
startup = win32process.STARTUPINFO()
(hProcess, hThread, processId, threadId) = win32process.CreateProcess(None, command, None, None, True, win32process.CREATE_BREAKAWAY_FROM_JOB, None, None, startup)

hJob = win32job.CreateJobObject(None, '')
extended_info = win32job.QueryInformationJobObject(hJob, win32job.JobObjectExtendedLimitInformation)
extended_info['BasicLimitInformation']['LimitFlags'] = win32job.JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE
win32job.SetInformationJobObject(hJob, win32job.JobObjectExtendedLimitInformation, extended_info)
win32job.AssignProcessToJobObject(hJob, hProcess)

Lorettelorgnette

+1: it seems like the most robust solution that works even if the parent process crashed. Here's the explanation on how it works – Serving

Answer 4

This is a hard thing to do. Windows does not actually store a process tree in the process space. Nor is it possible to terminate a process and specify that it's children should also die.

One way around that is to use taskkill and tell it to wack the whole tree.

Another way to do it (assuming that you are spawning the top-level process) is to use a module that was developed with this sort of thing in mind: http://benjamin.smedbergs.us/blog/tag/killableprocess/

In order to do this generically for yourself, you have to spend some time building the list backwards. That is, a process stores pointers to it's PARENT, but parents appear to not store information about children.

So you have to look at all the processes in the system (which really isn't that hard), and then manually connect the dots yourself by looking at the parent process field. Then, you select the tree you are interested in and walk the whole thing, killing each node in turn, one by one.

Note that Windows doesn't update a child's parent pointer when the parent dies, so there may be gaps in your tree. I'm not aware of anything you can do about those.

Illdefined

Answer 5

Put the children in a NT Job object, then you can kill all children

Philender

Job objects seems like the right way to approach this problem; too bad that this isn't integrated with the subprocess module. – Twocycle

Answer 6

I had the same problem and just killing process via windows command with option for child killing "/T"

def kill_command_windows(pid):
    '''Run command via subprocess'''
    dev_null = open(os.devnull, 'w')
    command = ['TASKKILL', '/F', '/T', '/PID', str(pid)]
    proc = subprocess.Popen(command, stdin=dev_null, stdout=sys.stdout, stderr=sys.stderr)

Superfuse

I was attempting to kill a Gradle build process kicked off with subprocess.Popen(). A simple process.terminate() or process.kill() did not work on WIndows 7, and neither did the psutils option above, but this did. – Nympha

Answer 7

I used kevin-smyth's answer to create a drop-in replacement for subprocess.Popen that confines the created child process in an anonymous job object, set up for terminating on close:

# coding: utf-8

from subprocess import Popen
import subprocess
import win32job
import win32process
import win32api


class JobPopen(Popen):
    """Start a process in a new Win32 job object.

    This `subprocess.Popen` subclass takes the same arguments as Popen and
    behaves the same way. In addition to that, created processes will be
    assigned to a new anonymous Win32 job object on startup, which will
    guarantee that the processes will be terminated by the OS as soon as
    either the Popen object, job object handle or parent Python process are
    closed.
    """

    class _winapijobhandler(object):
        """Patches the native CreateProcess function in the subprocess module
        to assign created threads to the given job"""

        def __init__(self, oldapi, job):
            self._oldapi = oldapi
            self._job = job

        def __getattr__(self, key):
            if key != "CreateProcess":
                return getattr(self._oldapi, key)  # Any other function is run as before
            else:
                return self.CreateProcess  # CreateProcess will call the function below

        def CreateProcess(self, *args, **kwargs):
            hp, ht, pid, tid = self._oldapi.CreateProcess(*args, **kwargs)
            win32job.AssignProcessToJobObject(self._job, hp)
            win32process.ResumeThread(ht)
            return hp, ht, pid, tid

    def __init__(self, *args, **kwargs):
        """Start a new process using an anonymous job object. Takes the same arguments as Popen"""

        # Create a new job object
        self._win32_job = self._create_job_object()

        # Temporarily patch the subprocess creation logic to assign created
        # processes to the new job, then resume execution normally.
        CREATE_SUSPENDED = 0x00000004
        kwargs.setdefault("creationflags", 0)
        kwargs["creationflags"] |= CREATE_SUSPENDED
        try:
            _winapi = subprocess._winapi  # Python 3
            _winapi_key = "_winapi"
        except AttributeError:
            _winapi = subprocess._subprocess  # Python 2
            _winapi_key = "_subprocess"
        try:
            setattr(subprocess, _winapi_key, JobPopen._winapijobhandler(_winapi, self._win32_job))
            super(JobPopen, self).__init__(*args, **kwargs)
        finally:
            setattr(subprocess, _winapi_key, _winapi)

    def _create_job_object(self):
        """Create a new anonymous job object"""
        hjob = win32job.CreateJobObject(None, "")
        extended_info = win32job.QueryInformationJobObject(hjob, win32job.JobObjectExtendedLimitInformation)
        extended_info['BasicLimitInformation']['LimitFlags'] = win32job.JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE
        win32job.SetInformationJobObject(hjob, win32job.JobObjectExtendedLimitInformation, extended_info)
        return hjob

    def _close_job_object(self, hjob):
        """Close the handle to a job object, terminating all processes inside it"""
        if self._win32_job:
            win32api.CloseHandle(self._win32_job)
            self._win32_job = None

    # This ensures that no remaining subprocesses are found when the process
    # exits from a `with JobPopen(...)` block.
    def __exit__(self, exc_type, value, traceback):
        super(JobPopen, self).__exit__(exc_type, value, traceback)
        self._close_job_object(self._win32_job)

    # Python does not keep a reference outside of the parent class when the
    # interpreter exits, which is why we keep it here.
    _Popen = subprocess.Popen  
    def __del__(self):
        self._Popen.__del__(self)
        self._close_job_object(self._win32_job)

Surbase

This is the only solution that worked for me after trying more about ~15 suggestions. Instantiate a subprocess using the JobPopen like this: with open(JobPopen(command, stdout=subprocess.PIPE, shell=True)) as p: # YOUR CODE HERE # And then, at the moment the script exits the 'with open' block - it will close that process. – Iconoduly

Answer 8

None of the answers will work if, like me, you're creating an Object from within your vbs script with logic similar to:

Set oExcel = CreateObject("Excel.Application")

This is because in Windows this spawns excel as a service and it has the owner of svchost.exe rather than your VBS script. Thank you to spokes for their answer which was surpremely helpful in diagnosing this.

I went about this fairly crudely and basically created a list of excel processes before I start it, then get another list of the excel processes afterwards and compare them, with the new PID being my new excel script. Then if I need to kill it, I kill it by identifying it through its PID.

this_script_pid = 0

try:
    running_excel_pids = [pid for pid in psutil.pids() \
                          if psutil.Process(pid).name() == "EXCEL.EXE"] # record all instances of excel before this starts
    p = subprocess.Popen(<starts vbs script that starts excel>)
    
    time.sleep(0.05) # give the script time to execute
    for pid in [p for p in psutil.pids() if psutil.Process(p).name() == "EXCEL.EXE"]:
        if pid not in running_excel_pids:
            this_script_pid = pid
            break
    
    p.communicate() # required to make our python program wait for the process to end


except:
    p.terminate() # kill the parent script
    if this_script_pid != 0:
        print("killing individual script")
        psutil.Process(this_script_pid).kill()

    else: 
        for pid in [p for p in psutil.pids() if psutil.Process(p).name() == "EXCEL.EXE"]:
            if (pid not in running_excel_pids) and (psutil.Process(pid).parent().name()=="svchost.exe"):
                proc = psutil.Process(pid)
                proc.kill()
    exit()  # gracefully quit

Note the above only works in my specific circumstance and even though I've tried to make it as targeted as possible it almost certainly should not be used in a multi-threaded environment.

The 0.05s wait was found empirically. 0.01s was too short, 0.03 worked, so 0.05s seemed safe.

The else in the except block is just a catch all in case it didn't manage to note which PID was created. It will kill all excel processes that have started as a service since the script began.

Neater answer is probably extending on Spokes' linked answer and running excel from within a shell, but I didn't have time to work through that.

Anis