How to terminate a python subprocess launched with shell=True

Question

I'm launching a subprocess with the following command:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

However, when I try to kill using:

p.terminate()

or

p.kill()

The command keeps running in the background, so I was wondering how can I actually terminate the process.

Note that when I run the command with:

p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)

It does terminate successfully when issuing the p.terminate().

Liven

What does your cmd look like? It might contain a command which triggers several processes to be started. So it’s not clear which process you talk about. – Mackle

related: Python: how to kill child process(es) when parent dies? – Awesome

does not having shell=True make a big difference? – Darladarlan

Answer 1

Use a process group so as to enable sending a signal to all the process in the groups. For that, you should attach a session id to the parent process of the spawned/child processes, which is a shell in your case. This will make it the group leader of the processes. So now, when a signal is sent to the process group leader, it's transmitted to all of the child processes of this group.

Here's the code:

import os
import signal
import subprocess

# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before  exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE, 
                       shell=True, preexec_fn=os.setsid) 

os.killpg(os.getpgid(pro.pid), signal.SIGTERM)  # Send the signal to all the process groups

Weft

@PiotrDobrogost: Sadly no, because os.setsid is not available in windows (docs.python.org/library/os.html#os.setsid), i don't know if this can help but you can look here (bugs.python.org/issue5115) for some insight about how to do it. – Weft

How does subprocess.CREATE_NEW_PROCESS_GROUP relate to this? – Risarise

@PiotrDobrogost: Well found :), apparently if you use the subprocess.CREATE_NEW_PROCESS_GROUP flag you can create a process group in Window check here for how: hg.python.org/cpython/file/321414874b26/Lib/test/… , sadly i don't have a windows machine in my hand so can you please try it and let me know ? :) – Weft

Running python -c "import subprocess; subprocess.Popen(['ping', '-t', 'google.com'], shell=True).terminate()" kills the subprocess. However I think it has nothing to do with subprocess.CREATE_NEW_PROCESS_GROUP as it's not being set anywhere in subprocess.py. Besides according to docs Process groups are used by the GenerateConsoleCtrlEvent function to enable sending a CTRL+BREAK signal to a group of console processes. and Popen.terminate() doesn't send any signal but calls TerminateProcess() Windows API function. – Risarise

our testing sugggests that setsid != setpgid, and that os.pgkill only kills subprocesses that still have the same process group id. processes that have changed process group are not killed, even though they may still have the same session id... – Kathiekathleen

@PiotrDobrogost: CREATE_NEW_PROCESS_GROUP can be used to emulate start_new_session=True on Windows – Awesome

This works even if shell=False. The finding of hwjp still applies. – Mailemailed

I would not recommend doing os.setsid(), since it has other effects as well. Among others, it disconnects the controlling TTY and makes the new process a process group leader. See win.tue.nl/~aeb/linux/lk/lk-10.html – Fagin

@parasietje: isn't this the whole point of this approach? To create a new process group, in which all processes can be killed with one signal? I had problems with a process that started a new process which I couldn't terminate. This answer solved my problem. – Pilsen

setsid creates a new session, while setpgrp only creates a new session if there is no session. if you nest several scripts, the outermost should call setsid and all others setpgrp otherwise the inner scripts will again reparent to init and not be killed automatically. en.wikipedia.org/wiki/Process_group#Details – Statutory

How would you do this in Windows? setsid is only available on *nix systems. – Osana

Will this fail where os.setsid() will fail with errno.EPERM (errno 1) such as running the script in a GUI terminal session? Or does it work because it happens after fork() but before exec() ? – Hydrophilous

This prints Terminated output, how to prevent it to be printed? @Weft – Hanni

if I don't have shell=True, how much does the answer change? – Darladarlan

couldn't preexec_fn=os.setpgrp be used in this case? – Halhalafian

@CharlieParker If shell=False, then I believe you can simply call the terminate() or kill() methods on the Popen object, as the question creator mentioned. – Tuantuareg

Doesn't work on MacOS. Solution from @Jovik with killing of all child processes does work. – Conversation

Answer 2

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
p.kill()

p.kill() ends up killing the shell process and cmd is still running.

I found a convenient fix this by:

p = subprocess.Popen("exec " + cmd, stdout=subprocess.PIPE, shell=True)

This will cause cmd to inherit the shell process, instead of having the shell launch a child process, which does not get killed. p.pid will be the id of your cmd process then.

p.kill() should work.

I don't know what effect this will have on your pipe though.

Hortatory

Nice and light solution for *nix, thanks! Works on Linux, should work for Mac as well. – Stemson

Very nice solution. If your cmd happens to be a shell script wrapper for something else, do call the final binary there with exec too in order to have only one subprocess. – Florio

This is beautiful. I have been trying to figure out how to spawn and kill a subprocess per workspace on Ubuntu. This answer helped me. Wish i could upvote it more than once – Gannie

this doesn't work if a semi-colon is used in the cmd – Strikebound

@speedyrazor - Does not work on Windows10. I think os specific answers should be clearly marked as such. – Kenley

Can someone tell if this has any bad effects? This does solve the problem. – Encumbrance

For some reason it kills my Putty session when I just type "exec ls" :D – Checkrein

Answer 3

If you can use psutil, then this works perfectly:

import subprocess

import psutil


def kill(proc_pid):
    process = psutil.Process(proc_pid)
    for proc in process.children(recursive=True):
        proc.kill()
    process.kill()


proc = subprocess.Popen(["infinite_app", "param"], shell=True)
try:
    proc.wait(timeout=3)
except subprocess.TimeoutExpired:
    kill(proc.pid)

Richelieu

AttributeError: 'Process' object has no attribute 'get_children for pip install psutil. – Maraud

I think get_children() should be children(). But it did not work for me on Windows, the process is still there. – Persas

@Persas - psutil API has changed and you're right: children() does the same thing as get_children() used to. If it doesn't work on Windows, then you might want to create a bug ticket in GitHub – Richelieu

this does not work if child X creates child SubX during calling proc.kill() for child A – Tryout

For some reason the other solutions would not work for me after many attempts, but this one did!! – Hebraize

this is a solution that will work on windows too – Goldstein

It works. but remember set shell=False – Alkmaar

Answer 4

I could do it using

from subprocess import Popen

process = Popen(command, shell=True)
Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))

it killed the cmd.exe and the program that i gave the command for.

(On Windows)

Shod

Or use the process name: Popen("TASKKILL /F /IM " + process_name), if you don't have it, you can get it from the command parameter. – Brause

Answer 5

When shell=True the shell is the child process, and the commands are its children. So any SIGTERM or SIGKILL will kill the shell but not its child processes, and I don't remember a good way to do it. The best way I can think of is to use shell=False, otherwise when you kill the parent shell process, it will leave a defunct shell process.

Pedicab

In order to avoid zombie processes one can terminate the process then wait and then poll the process to be sure that they are terminated, as described by "@SomeOne Maybe" in the following stackoverflow answer: #2761152 – Basset

Answer 6

None of these answers worked for me so I'm leaving the code that did work. In my case even after killing the process with .kill() and getting a .poll() return code the process didn't terminate.

Following the subprocess.Popen documentation:

"...in order to cleanup properly a well-behaved application should kill the child process and finish communication..."

proc = subprocess.Popen(...)
try:
    outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
    proc.kill()
    outs, errs = proc.communicate()

In my case I was missing the proc.communicate() after calling proc.kill(). This cleans the process stdin, stdout ... and does terminate the process.

Heave

This solution doesn't work for me in linux and python 2.7 – Hydrophobia

@xyz It did work for me in Linux and python 3.5. Check the docs for python 2.7 – Heave

@espinal, thanks, yes. It's possibly a linux issue. It's Raspbian linux running on a Raspberry 3 – Hydrophobia

@Weft and Bryant answers are ok in my case if you use as you said the communicate after calling kill. Thanks. – Spiderwort

This worked for me in Linux – Firecrest

Answer 7

As Sai said, the shell is the child, so signals are intercepted by it -- best way I've found is to use shell=False and use shlex to split the command line:

if isinstance(command, unicode):
    cmd = command.encode('utf8')
args = shlex.split(cmd)

p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Then p.kill() and p.terminate() should work how you expect.

Apoenzyme

In my case it doesn't really help given that cmd is "cd path && zsync etc etc". So that actually makes the command to fail! – Liven

Use absolute paths instead of changing directories... Optionally os.chdir(...) to that directory... – Apoenzyme

The ability to change the working directory for the child process is built-in. Just pass the cwd argument to Popen. – Odaniel

I used shlex, but still the issue persists, kill is not killing the child processes. – Alice

Answer 8

Send the signal to all the processes in group

    self.proc = Popen(commands, 
            stdout=PIPE, 
            stderr=STDOUT, 
            universal_newlines=True, 
            preexec_fn=os.setsid)

    os.killpg(os.getpgid(self.proc.pid), signal.SIGHUP)
    os.killpg(os.getpgid(self.proc.pid), signal.SIGTERM)

Spew

Answer 9

There is a very simple way for Python 3.5+ (actually tested on Python 3.8).

import subprocess, signal, time
p = subprocess.Popen(['cmd'], shell=True)
time.sleep(5) #Wait 5 secs before killing
p.send_signal(signal.CTRL_C_EVENT)

Then, your code may crash at some point if you have a keyboard input detection, or something like this. In this case, on the line of code/function where the error is given, just use:

try:
    FailingCode #here goes the code which is raising KeyboardInterrupt
except KeyboardInterrupt:
    pass

What this code is doing is just sending a "CTRL+C" signal to the running process, what will cause the process to get killed.

Odawa

Note: this is Wndows-specific. There is no CTRL_C_EVENT defined in Mac or Linux implementations of signal. Some alternative code (which I have not tested) can be found here. – Girhiny

Answer 10

Solution that worked for me

if os.name == 'nt':  # windows
    subprocess.Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))
else:
    os.kill(process.pid, signal.SIGTERM)

Audwin

Answer 11

Full blown solution that will kill running process (including subtree) on timeout reached or specific conditions via a callback function. Works both on windows & Linux, from Python 2.7 up to 3.12, and pypy 3.6 to pypy 3.10 as of this writing, see https://github.com/netinvent/command_runner for more info

Install with pip install command_runner

Example for timeout:

from command_runner import command_runner

# Kills ping after 2 seconds
exit_code, output = command_runner('ping 127.0.0.1', shell=True, timeout=2)

Example for specific condition: Here we'll stop ping if current system time seconds digit is > 5

from time import time
from command_runner import command_runner

def my_condition():
    # Arbitrary condition for demo
    return True if int(str(int(time()))[-1]) > 5

# Calls my_condition() every second (check_interval) and kills ping if my_condition() returns True
exit_code, output = command_runner('ping 127.0.0.1', shell=True, stop_on=my_condition, check_interval=1)

Osmious

For anyone getting this far down, command_runner uses psutil for this. The answer by @Richelieu (https://mcmap.net/q/25579/-how-to-terminate-a-python-subprocess-launched-with-shell-true) is a trimmed-down version of this. – Boston

Indeed, command_runner uses psutil. But it's code a bit more complex, using signals in order to avoid race conditions when using pids, and being 100% platform agnostic. Sure, the code can be trimmed down, but it will be less portable and more error prone. – Osmious