Node child processes: how to intercept signals like SIGINT

Question

In my Node app, I'm hooking on the SIGINT signal in order to gracefully stop (using pm2, but this is not relevant here).

My app also execs/spawns a couple of child processes.

I am able to hook on SIGINT to intercept it and perform graceful stop, however my child processes are passed through the same signal, and thus, instantly killed.

How can I intercept the SIGINT signal on my child processes?

A sample of what I'm doing:

const child = child_process.spawn('sleep', ['10000000']);
console.log(`Child pid: ${child.pid}`);

child.on('exit', (code, signal) => { console.log('Exit', code, signal); });

process.on('SIGINT', () => {
    console.log("Intercepting SIGINT");
});

Telpherage

Answer 1

By default, child processes created by child_process.spawn() have the same process group as the parent, unless they were called with the {detached:true} option.

The upshot is that this script will behave differently in different environments:

// spawn-test.js
const { spawn } = require('child_process');
const one = spawn('sleep', ['101']);
const two = spawn('sleep', ['102'], {detached: true});
two.unref();
process.on('SIGINT', function () {
  console.log('just ignore SIGINT');
});

On interactive shells, a SIGINT from Ctl-C is sent to the whole group by default, so the non-detached child will get the SIGINT and exit:

you@bash $ node spawn-test.js
^Cjust ignore SIGINT
# the parent process continues here, let's check children in another window:
you@bash [another-terminal-window] $ ps aux | grep sleep
... sleep 102
# note that sleep 101 is not running anymore
# because it recieved the SIGINT from the Ctl-C on parent

...but calls to kill(2) can just signal your parent process, so children stay alive:

you@bash $ node spawn-test.js & echo $?
[2] 1234
you@bash [another-terminal-window] $ kill -SIGINT 1234
you@bash [another-terminal-window] $ ps aux | grep sleep
... sleep 101
... sleep 102
# both are still running

However, pm2 is a whole other beast. Even if you try the above techniques, it kills the whole process tree, including your detached process, even with a long --kill-timeout:

# Test pm2 stop
you@bash $ pm2 start spawn-test.js --kill-timeout 3600
you@bash $ pm2 stop spawn-test
you@bash $ ps aux | grep sleep
# both are dead

# Test pm3 reload
you@bash $ pm2 start spawn-test.js --kill-timeout 3600
you@bash $ pm2 reload spawn-test
you@bash $ ps aux | grep sleep
# both have different PIDs and were therefore killed and restarted

This seems like a bug in pm2.

I've gotten around similar problems by using the init system (systemd in my case) rather than pm2, since this allows for greater control over signal handling.

On systemd, signals are sent to the whole group by default, but you can use KillMode=mixed to have the signal sent to the parent process only, but still SIGKILL child processes if they run beyond the timeout.

My systemd unit files look like this:

[Unit]
Description=node server with long-running children example

[Service]
Type=simple
Restart=always
RestartSec=30
TimeoutStopSec=3600
KillMode=mixed
ExecStart=/usr/local/bin/node /path/to/your/server.js

[Install]
WantedBy=multi-user.target

Reaper

Thanks for your great answer! – Telpherage

Nice answer, though it might be confusing for readers that your parent process signal handler does nothing to ignore the signal and allows the program to exit, but emits a string that indicates that you maybe intend for it to ignore the signal. – Orelia

@StevenLu, I think I see what you're saying here. Handling the signal prevents the default behavior (program exiting), so after the Ctl-C the program is actually still running, but maybe my console example implies that it exits. Notice that the next line has a comment on the bash prompt that says that it is in another terminal window. I assume this is not clear enough. Am I on the right track here? Maybe I can break up the example between two code blocks to make it more clear. – Reaper

@StevenLu, I just edited the first bash block, let me know if it's more clear now. – Reaper

Answer 2

Normally in C, you'd solve this by ignoring the signal in the child (or by spawning it in a new process group so that the terminal generated signal for the foreground process group doesn't reach it).

From looking at https://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options, it doesn't look like NodeJs exposes an API for this, however, it does have an option for spawning the child process through the shell, so what you can do is turn it on and ignore the signal in the shell, which will cause its ignored status to be inherited to the shell's children.

const child_process = require('child_process')
//const child = child_process.spawn('sleep', ['10000000']);
const child = child_process.spawn("trap '' INT; sleep 10000000", [], {shell: true });
console.log(`Child pid: ${child.pid}`);

child.on('exit', (code, signal) => { console.log('Exit', code, signal); });

process.on('SIGINT', () => {
    console.log("Intercepting SIGINT");
});


//emulate cat to keep the process alive
process.stdin.pipe(process.stdout);

Now when you press Ctrl-C, the Node process handles it and the sleep process lives on. (In case you're unfamiliar with the other terminal generated signals, you can easily kill this group by pressing Ctrl-\ (sends SIGQUIT to the group) if you don't mind the coredump).

Restitution

Answer 3

On pm2 has an option to disable to kill all tree process even using a detached child process, it's not documented well but if you run pm2 --help, you will notice

--no-treekill  Only kill the main process, not detached children

or if you are using ecosystem.config.js

treekill: false,

see https://github.com/Unitech/pm2/blob/master/lib/API/schema.json#L314, this option was added on 2015 https://github.com/Unitech/pm2/pull/1395

hope this info be useful for someone else facing the same situation.

Teratology