How do I kill background processes / jobs when my shell script exits?

Question

I am looking for a way to clean up the mess when my top-level script exits.

Especially if I want to use set -e, I wish the background process would die when the script exits.

Watchful

@DanielKaplan Try e.g. p=$(bash -c 'sleep 2 >/dev/null & echo $!'); sleep 1; ps -f -p "$p" to see that sleep 2 command is still running after bash has exited. – Duky

@DanielKaplan The sleep 2 command is running in background as a separate process; its command ends with &. – Duky

@Duky Apologies. I was incorrect about my first comment so I've deleted my others. – Limen

Answer 1

To clean up some mess, trap can be used. It can provide a list of stuff executed when a specific signal arrives:

trap "echo hello" SIGINT

but can also be used to execute something if the shell exits:

trap "killall background" EXIT

It's a builtin, so help trap will give you information (works with bash). If you only want to kill background jobs, you can do

trap 'kill $(jobs -p)' EXIT

Watch out to use single ', to prevent the shell from substituting the $() immediately.

Backpedal

then how do you killall child only ? (or am I missing something obvious) – Watchful

killall kills your children, but not you – Rodrich

kill $(jobs -p) doesn't work in dash, because it executes command substitution in a subshell (see Command Substitution in man dash) – Neolamarckism

is killall background supposed to be a placeholder? background is not in the man page... – Circumnavigate

@Neolamarckism but you could redirect the output of jobs -p to a temporary file and read it from there for kill. – Duky

kill $(jobs -p) is good, but prints usage info for 'kill' when there are no background jobs. IMHO, the best way for bash is jobs -p | xargs -r kill – Cretaceous

Some shells do not trigger EXIT on ctrl-c. Adding trap "exit" INT TERM ERR along with trap "kill 0" EXIT fixes this problem – Marquise

Answer 2

This works for me (improved thanks to the commenters):

trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

• kill -- -$$ sends a SIGTERM to the whole process group, thus killing also descendants. The <PGID> in kill -- -<PGID> is the group process id, which often, but not necessarily, is the PID that $$ variable contains. The few times PGID and PID differ you can use ps and other similar tools you can obtain the PGID, in your script.

  For example: pgid="$(ps -o pgid= $$ | grep -o '[0-9]*')" stores PGID in $pgid.

• Specifying signal EXIT is useful when using set -e (more details here).

Indianapolis

Should work well on the whole, but the child processes may change process groups. On the other hand it doesn't require job control, and may also get some grandchild processes missed by other solutions. – Noonan

I don't quite understand -$$. It evaluates to '-<PID>` eg -1234. In the kill manpage // builtin manpage a leading dash specifies the signal to be sent. However -- probably blocks that, but then the leading dash is undocumented otherwise. Any help? – Circumnavigate

@EvanBenn: Check man 2 kill, which explains that when a PID is negative, the signal is sent to all processes in the process group with the provided ID (en.wikipedia.org/wiki/Process_group). It's confusing that this is not mentioned in man 1 kill or man bash, and could be considered a bug in the documentation. – Whatever

kill -- -$$ might not terminate anything depending on how your script is called, e.g. if you call it like (sleep 100 & your_script) in shell. On the other hand, if you use kill -- 0 in the trap, it would terminate sleep 100 and the enclosing shell, too. – Duky

What if one needs to execute some cleanup after the child processes are terminated? I guess that instead of de-registering the trap, it would be possible to pass an empty string to it, so that SIGTERM is ignored: trap 'trap " " SIGTERM; kill 0; wait; cleanup SIGINT SIGTERM. Do you see any problem with this? – Satterwhite

When I use ShellCheck on this, it says SC2064: Use single quotes, otherwise this expands now rather than when signalled., should the answer be updated to reflect this? – Builtin

@EgidioDocile yes, kill 0 would not have an effect. – Duky

@MichaelYoo no, because the value of $$ remains the same. – Duky

Why do we have two nested traps here? – Isometry

@MohammedNoureldin The inner trap - SIGTERM will reset the current script SIGTERM response to the default kill behavior. Then, when kill -- -$$ is executed, the current script will receive SIGTERM and exit normally. – Puton

Should this expression maybe use single quotes instead of the double quotes? – Kermitkermy

Warning: you may need to disable the builtin shell kill command (enable -n kill) or use /bin/kill, as it may be that the builtin kill doesn't support the -pgid syntax. – Lurie

Where does this go in relation to the top-level script? – Facsimile

@Facsimile This tripped me up too. The trap command should go at the top of the script. I had it at the bottom. I probably would have discovered this if I had actually read the man page. – Styles

@MartijnPieters am using bash 5.2.15 and kill -SIGTERM -- -pid appears to be wroking just fine. – Valdis

@laur: I actually had to go to the bash source code to try and figure this out, as I can't remember the exact context I was working in when I discovered I had to disable the built-in kill command. Via the github bash mirror I learned that Bash 2.05 fixed support for -pid. So either I was working with a really ancient bash system, or perhaps it was a mac with some neutered bash version, or I wasn't working with bash at all. – Lurie

This is top notch but while using Asher256's backtrace (gist.github.com/Asher256/4c68119705ffa11adb7446f297a7beae) I had to apend || true at the end or would get error during Dockerfile build. No process to kill due to docker triggering it I suppose. As a side note putting custom exit logic before calling trap i.e. myExit; trap... works fine for me. – Rennes

Answer 3

Update: https://mcmap.net/q/108000/-how-do-i-kill-background-processes-jobs-when-my-shell-script-exits improves this by adding exit status and a cleanup function.

trap "exit" INT TERM
trap "kill 0" EXIT

Why convert INT and TERM to exit? Because both should trigger the kill 0 without entering an infinite loop.

Why trigger kill 0 on EXIT? Because normal script exits should trigger kill 0, too.

Why kill 0? Because nested subshells need to be killed as well. This will take down the whole process tree.

Numskull

This is the only version that worked for me on OSX with GNU bash, version 4.3.33(1)-release (x86_64-apple-darwin14.0.0). – Beebread

The only properly solution for me on OSX – Darwin

The only solution for my case on Debian. – Noelyn

Neither the answer by Johannes Schaub nor the one provided by tokland managed to kill the background processes my shell script started (on Debian). This solution worked. I don't know why this answer is not more upvoted. Could you expand more about what exactly kill 0 means/does? – Intermolecular

@Intermolecular if you have yet found out, here's an explanation for kill 0 – Spadework

This is awesome, but also kills my parent shell :-( – Consciousness

This solution is literally overkill. kill 0 (inside my script) ruined my whole X session! Perhaps in some cases kill 0 can be useful, but this does not change the fact that it is not general solution and should be avoided if possible unless there is very good reason to use it. It would be nice to add a warning that it may kill parent shell or even whole X session, not just background jobs of a script! – Adest

Dumb workaround that restricts the kill 0 to just the shell script's processes: run the script under setsid -w (which has other side effects, unfortunately); #6550163 – Hubblebubble

I would add QUIT as well to the traps. – Pahari

Thank you. For me this improved it if my script just started a background process and stopped trap "exit" INT TERM trap "sleep 0.1; kill 0" EXIT – Enthronement

While this might be an interesting solution under some circumstances, as pointed out by @Consciousness this will kill the whole process group which includes the launching process (i.e. the parent shell in most cases). Definitely not something you want when you are running a script via an IDE. – Baroja

This is the only answer that worked in my case. – Marylinmarylinda

Answer 4

The trap 'kill 0' SIGINT SIGTERM EXIT solution described in @tokland's answer is really nice, but latest Bash crashes with a segmentation fault when using it. That's because Bash, starting from v. 4.3, allows trap recursion, which becomes infinite in this case:

1. shell process receives SIGINT or SIGTERM or EXIT;
2. the signal gets trapped, executing kill 0, which sends SIGTERM to all processes in the group, including the shell itself;
3. go to 1 :)

This can be worked around by manually de-registering the trap:

trap 'trap - SIGTERM && kill 0' SIGINT SIGTERM EXIT

The more fancy way that allows printing the received signal and avoids "Terminated:" messages:

#!/usr/bin/env bash

trap_with_arg() { # from https://mcmap.net/q/110297/-is-it-possible-to-detect-which-trap-signal-in-bash-duplicate
  local func="$1"; shift
  for sig in "$@"; do
    trap "$func $sig" "$sig"
  done
}

stop() {
  trap - SIGINT EXIT
  printf '\n%s\n' "received $1, killing child processes"
  kill -s SIGINT 0
}

trap_with_arg 'stop' EXIT SIGINT SIGTERM SIGHUP

{ i=0; while (( ++i )); do sleep 0.5 && echo "a: $i"; done } &
{ i=0; while (( ++i )); do sleep 0.6 && echo "b: $i"; done } &

while true; do read; done

UPD: added a minimal example; improved stop function to avoid de-trapping unnecessary signals and to hide "Terminated:" messages from the output. Thanks Trevor Boyd Smith for the suggestions!

Eelgrass

in stop() you provide the first argument as the signal number but then you hardcode what signals are being deregistered. rather than hardcode the signals being deregistered you could use the first argument to deregister in the stop() function (doing so would potentially stop other recursive signals (other than the 3 hardcoded)). – Radix

@TrevorBoydSmith, this would not work as expected, I guess. For example, the shell might be killed with SIGINT, but kill 0 sends SIGTERM, which will get trapped once again. This will not produce infinite recursion, though, because SIGTERM will be de-trapped during the second stop call. – Eelgrass

Probably, trap - $1 && kill -s $1 0 should work better. I'll test and update this answer. Thank you for the nice idea! :) – Eelgrass

Nope, trap - $1 && kill -s $1 0 woldn't work too, as we can't kill with EXIT. But it is really sufficient do de-trap TERM, because kill sends this signal by default. – Eelgrass

I tested recursion with EXIT, the trap signal-handler is always only executed once. – Radix

Updated the answer, now stop doesn't require modification, regardless of the signal list. And it seems that, when the shell gets killed with INT, it doesn't print "Terminated:" messages (which it does in case of TERM). – Eelgrass

@TrevorBoydSmith, I meant that you cannot kill -s EXIT pid, which stop will try to do if it was written like trap - $1 && kill -s $1 0 and invoked by EXIT. – Eelgrass

/bin/sh symlinked to dash produces the error trap: SIGINT: bad trap. Removing SIG prefix from signal names (such that SIGINT becomes INT, etc.) works as expected with both dash and bash. – Pitzer

Had I not know anything about bash, I would have assumed "recieved $1, killing children" meant "earning a dollar and murdering kids". You might wanna reword that. – Photima

@Photima done, now it should be harder to misinterpret the message. – Eelgrass

I don't understand why do we have two nested traps? – Isometry

@MohammedNoureldin the second trap - SIGINT EXIT command clears the trap so the stop function doesn't get called recursively when the process finally exits. – Eelgrass

Answer 5

trap 'kill $(jobs -p)' EXIT

I would make only minor changes to Johannes' answer and use jobs -pr to limit the kill to running processes and add a few more signals to the list:

trap 'kill $(jobs -pr)' SIGINT SIGTERM EXIT

Immiscible

Why not kill the stopped jobs, too? In Bash EXIT trap will be run in case of SIGINT and SIGTERM, too, so the trap would be called twice in case of such a signal. – Duky

This works!! The other answers kill the calling process as well as the subprocesses, which is an issue if a script is called directly by a desktop environment or is called by another program you want to keep alive. Thank you! – Alternation

Answer 6

To be on the safe side I find it better to define a cleanup function and call it from trap:

cleanup() {
        local pids=$(jobs -pr)
        [ -n "$pids" ] && kill $pids
}
trap "cleanup" INT QUIT TERM EXIT [...]

or avoiding the function altogether:

trap '[ -n "$(jobs -pr)" ] && kill $(jobs -pr)' INT QUIT TERM EXIT [...]

Why? Because by simply using trap 'kill $(jobs -pr)' [...] one assumes that there will be background jobs running when the trap condition is signalled. When there are no jobs one will see the following (or similar) message:

kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

because jobs -pr is empty - I ended in that 'trap' (pun intended).

Eventually

This test case [ -n "$(jobs -pr)" ] doesn't work on my bash. I use GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu). The "kill: usage" message keeps popping up. – Pavlov

I suspect it has to do with the fact that jobs -pr doesn't return the PIDs of the children of the background processes. It doesn't tear the entire process tree down, only trims off the roots. – Pavlov

Answer 7

function cleanup_func {
    sleep 0.5
    echo cleanup
}

trap "exit \$exit_code" INT TERM
trap "exit_code=\$?; cleanup_func; kill 0" EXIT

# exit 1
# exit 0

Like https://mcmap.net/q/108000/-how-do-i-kill-background-processes-jobs-when-my-shell-script-exits, but with added exit-code

Pah

Where does exit_code come from in INT TERM trap? – Duky

@Duky IIUC, the EXIT trap is always invoked when the script exits. It sets the global variable exit_code to the exit code of the last command executed. After running cleanup_func, it then sends SIGTERM "to every process in the process group of the calling process", including itself (see kill(2)). SIGTERM is trapped and exits with $exit_code. Now, if you press ^C during script execution, $exit_code in the INT trap will be empty and exit will be invoked with the exit code of the last command: 130 (see help exit). exit triggers the EXIT trap: start from the top. – Danelledanete

Answer 8

A nice version that works under Linux, BSD and MacOS X. First tries to send SIGTERM, and if it doesn't succeed, kills the process after 10 seconds.

KillJobs() {
    for job in $(jobs -p); do
            kill -s SIGTERM $job > /dev/null 2>&1 || (sleep 10 && kill -9 $job > /dev/null 2>&1 &)

    done
}

TrapQuit() {
    # Whatever you need to clean here
    KillJobs
}

trap TrapQuit EXIT

Please note that jobs does not include grand children processes.

Income

Answer 9

I made an adaption of @tokland's answer combined with the knowledge from http://veithen.github.io/2014/11/16/sigterm-propagation.html when I noticed that trap doesn't trigger if I'm running a foreground process (not backgrounded with &):

#!/bin/bash

# killable-shell.sh: Kills itself and all children (the whole process group) when killed.
# Adapted from http://stackoverflow.com/a/2173421 and http://veithen.github.io/2014/11/16/sigterm-propagation.html
# Note: Does not work (and cannot work) when the shell itself is killed with SIGKILL, for then the trap is not triggered.
trap "trap - SIGTERM && echo 'Caught SIGTERM, sending SIGTERM to process group' && kill -- -$$" SIGINT SIGTERM EXIT

echo $@
"$@" &
PID=$!
wait $PID
trap - SIGINT SIGTERM EXIT
wait $PID

Example of it working:

$ bash killable-shell.sh sleep 100
sleep 100
^Z
[1]  + 31568 suspended  bash killable-shell.sh sleep 100

$ ps aux | grep "sleep"
niklas   31568  0.0  0.0  19640  1440 pts/18   T    01:30   0:00 bash killable-shell.sh sleep 100
niklas   31569  0.0  0.0  14404   616 pts/18   T    01:30   0:00 sleep 100
niklas   31605  0.0  0.0  18956   936 pts/18   S+   01:30   0:00 grep --color=auto sleep

$ bg
[1]  + 31568 continued  bash killable-shell.sh sleep 100

$ kill 31568
Caught SIGTERM, sending SIGTERM to process group
[1]  + 31568 terminated  bash killable-shell.sh sleep 100

$ ps aux | grep "sleep"
niklas   31717  0.0  0.0  18956   936 pts/18   S+   01:31   0:00 grep --color=auto sleep

Cassondra

Answer 10

I finally have found a solution that appears to work in all cases to kill all descents recursively regardless of whether they are jobs, or sub-processes. The other solutions here all seemed to fail with things such as:

while ! ffmpeg ....
do
  sleep 1
done

In my situation, ffmpeg would keep running after the parent script exited.

I found a solution here to recursively getting the PIDs of all child processes recursively and used that in the trap handler thus:

cleanup() {
    # kill all processes whose parent is this process
    kill $(pidtree $$ | tac)
}

pidtree() (
    [ -n "$ZSH_VERSION"  ] && setopt shwordsplit
    declare -A CHILDS
    while read P PP;do
        CHILDS[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)
    walk() {
        echo $1
        for i in ${CHILDS[$1]};do
            walk $i
        done
    }

    for i in "$@";do
        walk $i
    done
)

trap cleanup EXIT

The above placed at the start of a bash script succeeds in killing all child processes. Note that pidtree is called with $$ which is the PID of the bash script that is exiting and the list of PIDs (one per line) is reversed using tac to try and ensure that prarent processes are killed only after their children to avoid possible race conditions in loops such as the example I gave.

Mccourt

Answer 11

None of the answers here worked for me in the case of a continuous integration (CI) script that starts background processes from subshells. For example:

(cd packages/server && npm start &)

The subshell terminates after starting the background process, which therefore ends up with parent PID 1.

With PPID not an option, the only portable (Linux and MacOS) and generic (independent of process name, listening ports, etc.) approach left is the process group (PGID). However, I can't just kill that because it would kill the script process, which would fail the CI job.

# Terminate the given process group, excluding this process. Allows 2 seconds
# for graceful termination before killing remaining processes. This allows
# shutdown errors to be printed, while handling processes that fail to
# terminate quickly.
kill_subprocesses() {
  echo "Terminating subprocesses of PGID $1 excluding PID $$"
  # Get all PIDs in this process group except this process
  # (pgrep on NetBSD/MacOS does this by default, but Linux pgrep does not)
  # Uses a heredoc instead of piping to avoid including the grep PID
  pids=$(grep -Ev "\\<$$\\>" <<<"$(pgrep -g "$1")")
  if [ -n "$pids" ]; then
    echo "Terminating processes: ${pids//$'\n'/, }"
    # shellcheck disable=SC2086
    kill $pids || true
  fi
  sleep 2
  # Check for remaining processes and kill them
  pids=$(grep -Ev "\\<$$\\>" <<<"$(pgrep -g "$1")")
  if [ -n "$pids" ]; then
    echo "Killing remaining processes: ${pids//$'\n'/, }"
    # shellcheck disable=SC2086
    kill -9 $pids || true
  fi
}

# Terminate subprocesses on exit or interrupt
# shellcheck disable=SC2064
trap "kill_subprocesses $$" EXIT SIGINT SIGTERM

Closer

Answer 12

Another option is it to have the script set itself as the process group leader, and trap a killpg on your process group on exit.

EDIT: a possible bash hack to create a new process group is to use setsid(1) but only if we're not already the process group leader (can query it with ps).

Placing this at the beginning of the script can achieve that.

# Create a process group and exec the script as its leader if necessary
[[ "$(ps -o pgid= $$)" -eq "$$" ]] || exec setsid /bin/bash "$0" "$@"

Then signaling the process group with kill -- -$$ would work as expected even when script is not already the process group leader.

Rodrich

How do you set the process as process group leader? What is "killpg"? – Duky

killpg is C api to send signal (=kill) to a Process Group so is exactly what the kill -- -$$ and kill 0 answers suggest; starting a new proccess group is the novel idea here but needs details on how to do this from bash... – Tribute

setsid(1) can do it, and we can test whether we're the leader with ps. So the bash hack would be to add something like this to the beginning of the script ` [[ "$(ps -o pgid= $$)" -eq "$$" ]] || exec setsid /bin/bash "$0" "$@"` – Rodrich

Answer 13

jobs -p does not work in all shells if called in a sub-shell, possibly unless its output is redirected into a file but not a pipe. (I assume it was originally intended for interactive use only.)

What about the following:

trap 'while kill %% 2>/dev/null; do jobs > /dev/null; done' INT TERM EXIT [...]

The call to "jobs" is needed with Debian's dash shell, which fails to update the current job ("%%") if it is missing.

Noonan

Hmm interesting approach, but it does not seem to work. Consider scipt trap 'echo in trap; set -x; trap - TERM EXIT; while kill %% 2>/dev/null; do jobs > /dev/null; done; set +x' INT TERM EXIT; sleep 100 & while true; do printf .; sleep 1; done If you run it in Bash (5.0.3) and try to terminate, there seems to be an infinite loop. However, if you terminate it again, it works. Even by Dash (0.5.10.2-6) you have to terminate it twice. – Duky

Answer 14

Just for diversity I will post variation of https://mcmap.net/q/108000/-how-do-i-kill-background-processes-jobs-when-my-shell-script-exits , because that solution leads to message "Terminated" in my environment:

trap 'test -z "$intrap" && export intrap=1 && kill -- -$$' SIGINT SIGTERM EXIT

Arela

Answer 15

Universal solution which works also in sh (jobs there does not output anything to stdout):

trap "pkill -P $$" EXIT INT

Seville

This only kills child processes. It wouldn't handle common cases like jobs started by a subshell (which end up with PPID 1). Killing the process group with -g would do that. – Closer

Well, I assumed typical scenario when sub-processes are responsible for killing their children. – Seville

Answer 16

So script the loading of the script. Run a killall (or whatever is available on your OS) command that executes as soon as the script is finished.

Jene