Does linux kill background processes if we close the terminal from which it has started?

Question

I have an embedded system, on which I do telnet and then I run an application in background:

./app_name &

Now if I close my terminal and do telnet from other terminal and if I check then I can see this process is still running.

To check this I have written a small program:

#include<stdio.h>
main()
{
    while(1);
}

I ran this program in my local linux pc in background and I closed the terminal.

Now, when I checked for this process from other terminal then I found that this process was also killed.

My question is:

• Why undefined behavior for same type of process?
• On which it is dependent?
• Is it dependent on version of Linux?

Loganiaceous

Answer 1

Who should kill jobs?

Normally, foreground and background jobs are killed by SIGHUP sent by kernel or shell in different circumstances.

---

When does kernel send SIGHUP?

Kernel sends SIGHUP to controlling process:

• for real (hardware) terminal: when disconnect is detected in a terminal driver, e.g. on hang-up on modem line;
• for pseudoterminal (pty): when last descriptor referencing master side of pty is closed, e.g. when you close terminal window.

Kernel sends SIGHUP to other process groups:

• to foreground process group, when controlling process terminates;
• to orphaned process group, when it becomes orphaned and it has stopped members.

Controlling process is the session leader that established the connection to the controlling terminal.

Typically, the controlling process is your shell. So, to sum up:

• kernel sends SIGHUP to the shell when real or pseudoterminal is disconnected/closed;
• kernel sends SIGHUP to foreground process group when the shell terminates;
• kernel sends SIGHUP to orphaned process group if it contains stopped processes.

Note that kernel does not send SIGHUP to background process group if it contains no stopped processes.

---

When does bash send SIGHUP?

Bash sends SIGHUP to all jobs (foreground and background):

• when it receives SIGHUP, and it is an interactive shell (and job control support is enabled at compile-time);
• when it exits, it is an interactive login shell, and huponexit option is set (and job control support is enabled at compile-time).

See more details here.

Notes:

• bash does not send SIGHUP to jobs removed from job list using disown;
• processes started using nohup ignore SIGHUP.

More details here.

---

What about other shells?

Usually, shells propagate SIGHUP. Generating SIGHUP at normal exit is less common.

---

Telnet or SSH

Under telnet or SSH, the following should happen when connection is closed (e.g. when you close telnet window on PC):

1. client is killed;
2. server detects that client connection is closed;
3. server closes master side of pty;
4. kernel detects that master pty is closed and sends SIGHUP to bash;
5. bash receives SIGHUP, sends SIGHUP to all jobs and terminates;
6. each job receives SIGHUP and terminates.

---

Problem

I can reproduce your issue using bash and telnetd from busybox or dropbear SSH server: sometimes, background job doesn't receive SIGHUP (and doesn't terminate) when client connection is closed.

It seems that a race condition occurs when server (telnetd or dropbear) closes master side of pty:

1. normally, bash receives SIGHUP and immediately kills background jobs (as expected) and terminates;
2. but sometimes, bash detects EOF on slave side of pty before handling SIGHUP.

When bash detects EOF, it by default terminates immediately without sending SIGHUP. And background job remains running!

---

Solution

It is possible to configure bash to send SIGHUP on normal exit (including EOF) too:

• Ensure that bash is started as login shell. The huponexit works only for login shells, AFAIK.

  Login shell is enabled by -l option or leading hyphen in argv[0]. You can configure telnetd to run /bin/bash -l or better /bin/login which invokes /bin/sh in login shell mode.

  E.g.:

  telnetd -l /bin/login
  

• Enable huponexit option.

  E.g.:

  shopt -s huponexit
  

  Type this in bash session every time or add it to .bashrc or /etc/profile.

---

Why does the race occur?

bash unblocks signals only when it's safe, and blocks them when some code section can't be safely interrupted by a signal handler.

Such critical sections invoke interruption points from time to time, and if signal is received when a critical section is executed, it's handler is delayed until next interruption point happens or critical section is exited.

You can start digging from quit.h in the source code.

Thus, it seems that in our case bash sometimes receives SIGHUP when it's in a critical section. SIGHUP handler execution is delayed, and bash reads EOF and terminates before exiting critical section or calling next interruption point.

---

Reference

• "Job Control" section in official Glibc manual.
• Chapter 34 "Process Groups, Sessions, and Job Control" of "The Linux Programming Interface" book.

Threesquare

Perfect, Thanks for the answer – Loganiaceous

@Chirag, I've added a guess to the answer, why the race may occur in bash. – Threesquare

@Threesquare Thanks. However I did some local testing using trap. Bash receives SIGCLD and SIGCONT along with SIGHUP. Why? And why do we need to Ensure that bash is started as login shell. ? – Embryologist

@Embryologist SIGCLD is usually sent when a child process terminates. One case when SIGCONT is sent is when a process group becomes an orphaned process group. Not sure if it's the case, need more data. – Threesquare

@Embryologist "And why do we need to Ensure that bash is started as login shell. ?" -- huponexit works for login shells only IIRC https://mcmap.net/q/501507/-when-did-hup-stop-getting-sent-and-what-can-i-do-about-it "If the huponexit shell option has been set with shopt, bash sends a SIGHUP to all jobs when an interactive login shell exits." I'll update the answer. – Threesquare

@Threesquare Thanks for the clarification. Here is my full test case. https://paste.wentropy.com/zAcW – Embryologist

Answer 2

When you close the terminal, shell sends SIGHUP to all background processes – and that kills them. This can be suppressed in several ways, most notably:

nohup

When you run program with nohup it catches SIGHUP and redirect program output.

$ nohup app &

disown

disown tells shell not to send SIGHUP

$ app &
$ disown

Is it dependent on version of linux?

It is dependent on your shell. Above applies at least for bash.

Loadstone

Note than shell usually sends SIGHUP only when it receives SIGHUP itself (see my answer). The OP's problem, as I understand, is that the shell either does not receive or does not propagate SIGHUP. – Threesquare

Answer 3

AFAIK in both cases the process should be killed. In order to avoid this you have to issue a nohup like the following:

> nohup ./my_app &

This way your process will continue executing. Probably the telnet part is due to a BUG similar to this one:

https://bugzilla.redhat.com/show_bug.cgi?id=89653

Mouse

Answer 4

In order completely understand whats happening you need to get into unix internals a little bit.

When you are running a command like this

./app_name &

The app_name is sent to background process group. You can check about unix process groups here

When you close bash with normal exit it triggers SIGHUP hangup signal to all its jobs. Some information on unix job control is here.

In order to keep your app running when you exit bash you need to make your app immune to hangup signal with nohup utility.

nohup - run a command immune to hangups, with output to a non-tty

And finally this is how you need to do it.

nohup app_name & 2> /dev/null;

Devoe

Bash doesn't always (ever?) send a SIGHUP to all its jobs with normal exit. It WILL forward a SIGHUP from the terminal it's running in. See my answer to this question: serverfault.com/questions/117152/… . Strangely, I have yet to see bash send a SIGHUP even with shopt -s huponexit, at least in CentOS7.1. Neither did I see it when bash was run as /bin/sh, neither did I see it with /bin/csh. – Ribbonwood

Does the backgrounded linux process terminate via this signal if I run the background process from a Java ProcessBuilder? – Perryperryman

Answer 5

In modern Linux--that is, Linux with systemd--there is an additional reason this might happen which you should be aware of: "linger".

systemd kills processes left running from a login shell, even if the process is properly daemonized and protected from HUP. This is the default behavior in modern configurations of systemd.

If you run

loginctl enable-linger $USER

you can disable this behavior, allowing background processes to keep running. The mechanisms covered by the other answers still apply, however, and you should also protect your process against them.

enable-linger is permanent until it is re-disabled. You can check it with

ls /var/lib/systemd/linger

This may have files, one per username, for users who have enable-linger. Any user listed in the directory has the ability to leave background processes running at logout.

Spondee