Node.js spawning a child process interactively with separate stdout and stderr streams

Question

Consider the following C program (test.c):

#include <stdio.h>

int main() {
  printf("string out 1\n");
  fprintf(stderr, "string err 1\n");
  getchar();
  printf("string out 2\n");
  fprintf(stderr, "string err 2\n");
  fclose(stdout);
}

Which should print a line to stdout, a line to stderr, then wait for user input, then another line to stdout and another line to stderr. Very basic! When compiled and run on the command line the output of the program when complete (user input is received for getchar()):

$ ./test 
string out 1
string err 1

string out 2
string err 2

When trying to spawn this program as a child process using nodejs with the following code:

var TEST_EXEC = './test';

var spawn = require('child_process').spawn;
var test = spawn(TEST_EXEC);

test.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

test.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

// Simulate entering data for getchar() after 1 second
setTimeout(function() {
  test.stdin.write('\n');
}, 1000);

The output appears like this:

$ nodejs test.js 
stderr: string err 1

stdout: string out 1
string out 2

stderr: string err 2

Very different from the output as seen when running ./test in the terminal. This is because the ./test program isn't running in an interactive shell when spawned by nodejs. The test.c stdout stream is buffered and when run in a terminal as soon as a \n is reached the buffer is flushed but when spawned in this way with node the buffer isn't flushed. This could be resolved by either flushing stdout after every print, or changing the stdout stream to be unbuffered so it flushes everything immediately. Assuming that test.c source isn't available or modifiable, neither of the two flushing options mentioned can be implemented.

I then started looking at emulating an interactive shell, there's pty.js (pseudo terminal) which does a good job, for example:

var spawn = require('pty.js').spawn;
var test = spawn(TEST_EXEC);

test.on('data', function (data) {
  console.log('data: ' + data);
});

// Simulate entering data for getchar() after 1 second
setTimeout(function() {
  test.write('\n');
}, 1000);

Which outputs:

$ nodejs test.js
data: string out 1
string err 1

data: 

data: string out 2
string err 2

However both stdout and stderr are merged together (as you would see when running the program in a terminal) and I can't think of a way to separate the data from the streams.

So the question..

Is there any way using nodejs to achieve the output as seen when running ./test without modifying the test.c code? Either by terminal emulation or process spawning or any other method?

Cheers!

Walters

Thanks for putting a considerable amount of effort into the description of the problem! I had the same problem when running someone else's python script from within node, but didn't know anything about the underlying buffering mechanics. Apparently (my python skills are +/- nonexistent), simply using 'python -u' enables unbuffered IO and solved my problem! – Intelsat

@Walters Have you worked out this problem? I am facing the similar issue. You can it from #42131177 – Gereld

Answer 1

I was just revisiting this since there is now a 'shell' option available for the spawn command in node since version 5.7.0. Unfortunately there doesn't seem to be an option to spawn an interactive shell (I also tried with shell: '/bin/sh -i' but no joy). However I just found this which suggests using 'stdbuf' allowing you to change the buffering options of the program that you want to run. Setting them to 0 on everything produces unbuffered output for all streams and they're still kept separate.

Here's the updated javascript:

var TEST_EXEC = './test';

var spawn = require('child_process').spawn;
var test = spawn('stdbuf', ['-i0', '-o0', '-e0', TEST_EXEC]);

test.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

test.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

// Simulate entering data for getchar() after 1 second
setTimeout(function() {
  test.stdin.write('\n');
}, 1000);

Looks like this isn't pre-installed on OSX and of course not available for Windows, may be similar alternatives though.

Walters

That 'stdbuf' trick solved the problem I had with stdout buffering of a command running inside a spawned shell (using Node v0.10.29 on Raspbian Jessie). Thanks a lot, gratz! – Straticulate

Glad this helped @Straticulate – Walters

Answer 2

I tried the answer by user568109 but this does not work, which makes sense since the pipe only copies the data between streams. Hence, it only gets to process.stdout when the buffer is flushed... The following appears to work:

var TEST_EXEC = './test';

var spawn = require('child_process').spawn;
var test = spawn(TEST_EXEC, [], { stdio: 'inherit' });

//the following is unfortunately not working 
//test.stdout.on('data', function (data) {
//  console.log('stdout: ' + data);
//});

Note that this effectively shares stdio's with the node process. Not sure if you can live with that.

Intelsat

Using spawn() with option { stdio: 'inherit' } is indeed what enables interleaved output to the parent streams, but - at least as of 0.10.10 - you thereby lose the ability to "listen in" on the streams via events: From the doc re the .stdin, .stdout, .stderr properties on child-process objects: "If the child stdio streams are shared with the parent, then this will not be set. " nodejs.org/api/child_process.html#child_process_child_stdout. In other words: as of 0.10.10, your code fails when accessing the .stdout property. I'm happy to +1 your answer if you fix that. – Adlay

You're right! This sucks; I don't think there is something like a 'data' event on a writable stream like process.stdout, so I don't know whether you can get to that output in some way :-( – Intelsat

+1 for sharing the spawn() + { stdio: 'inherit' } technique (even though it doesn't solve the OP's problem). As for listening to writes: someone found a workaround for the process itself by 'subclassing' the process.stdout.write() method, but, sadly, it doesn't work for what child processes write to shared streams: gist.github.com/bsatrom/1349384 – Adlay

Answer 3

You can do this :

var TEST_EXEC = 'test';
var spawn = require('child_process').spawn;
var test = spawn(TEST_EXEC);

test.stdin.pipe(process.stdin);
test.stdout.pipe(process.stdout);
test.stderr.pipe(process.stderr);

When you use events on stdout and stderr to print the output on console.log, you will get jumbled output because of asynchronous execution of the functions. The output will be ordered for a stream independently, but output can still get interleaved among stdin,stdout and stderr.

Krasnoff

Unless I've missed something, how could I programmatically use the outputs in this way? – Walters

You can pipe the standard streams of 'test' to another process say process 'a'(here it is node itself), whose output will be exactly that of 'test' while still using your code to listen outputs of test. – Krasnoff

But to get stdout of test when a '\n' is reached, the process would need to be a terminal, or pseudo terminal (pty) to ensure test pipes would flush as if run in an interactive shell.. Either of those would mean losing the distinction between stdout and stderr like the examples above? – Walters

This would be very useful: github.com/joyent/node/issues/2754 And this is the goal - tricking an application into thinking it's running in an interactive terminal, just without the merged stdout and stderr: #1401502 – Walters

don't you mean process.stdin.pipe(test.stdin)? – Manche