spawning node.js child processes results in zombie processes on cloud foundry

Question

I have a node.js app, and I want to spawn child processes using the code listed below.

When I run this app locally, each of the 'ps' commands nicely trigger the close and exit events. On our cloud foundry (pivotal.io) app however, the stdout.close is triggered, but the 'close' and 'exit' events of the child process itself never. Also, the processes remain as zombie processes in memory (so after ~500 request the server dies on a E_SPAWN error). So it seems that the exit handler in the node.js process handle is never triggered, resulting in the exit code of the child process not being read.

Can this relate to the container warden, cgroups...? Does anybody have a solution for this or at least ran into the same issue?

The test code:

var cp = require('child_process');

//..create express app

app.get('/foo/', function(req, res, next) {
    var child = cp.spawn("ps",["aux"]);
    child.stderr.pipe(process.stderr);

    child.stdout.on('data', function(data) {
        console.log('data');
        res.send("\n<br>OUT" + data.toString());
    });

    child.stdout.on('close', function() {
        console.log('close stdout');
        res.send("\n<br>CLOSE STDOUT");
    });

    child.on('close', function() {
        console.log('close');
        res.send("\n<br>CLOSE");
    });

    child.on('exit', function() {
        console.log('exit');
        res.send("\n<br>EXIT");
    });
});

app.listen();

Example ps aux output:

<br>OUTUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   1124   308 ?        S<s  14:15   0:00 wshd: 189gaonkujh    
vcap        31  0.2  0.0 602676 21692 ?        S<sl 14:15   0:00 node index.js 1234
vcap        33  0.0  0.0      0     0 ?        Z<   14:17   0:00 [ps] <defunct>
vcap        34  0.0  0.0  15332  1172 ?        R<   14:17   0:00 ps aux

UPDATE

See the comments for a workaround: use a custom start command that starts with 'true;', e.g. cf push myapp -c 'true;node index.js'

Daladier

A similar issue is described at: pivotaltracker.com/story/show/83352380.The work around is to not rely on 'npm start' to start the process, but rather provide a custom start command to the CF container that starts with 'true;', for example cf push myapp -c 'true;node index.js' – Daladier

Here is an excellent explanation about what's going on: groups.google.com/a/cloudfoundry.org/forum/#!topic/vcap-dev/… – Cultism

Answer 1

You're not killing your child processes, so they're hanging out as zombies. Kill your zombies, by killing your children (yea, that sounds pretty weird) ..

child.on('exit', function() {
    console.log('exit');
    res.send("\n<br>EXIT");
    child.kill();
});

Allergic