Exec : display stdout "live"

Question

I have this simple script :

var exec = require('child_process').exec;

exec('coffee -cw my_file.coffee', function(error, stdout, stderr) {
    console.log(stdout);
});

where I simply execute a command to compile a coffee-script file. But stdout never get displayed in the console, because the command never ends (because of the -w option of coffee). If I execute the command directly from the console I get message like this :

18:05:59 - compiled my_file.coffee

My question is : is it possible to display these messages with the node.js exec ? If yes how ? !

Trifocal

I came here looking for capturing stdout from Python executable. Note that all of the below will work, but you need to run python with a "-u" option, to make outout unbuffered and thereby have live updates. – Cedric

Answer 1

Don't use exec. Use spawn which is an EventEmmiter object. Then you can listen to stdout/stderr events (spawn.stdout.on('data',callback..)) as they happen.

From NodeJS documentation:

var spawn = require('child_process').spawn,
    ls    = spawn('ls', ['-lh', '/usr']);

ls.stdout.on('data', function (data) {
  console.log('stdout: ' + data.toString());
});

ls.stderr.on('data', function (data) {
  console.log('stderr: ' + data.toString());
});

ls.on('exit', function (code) {
  console.log('child process exited with code ' + code.toString());
});

exec buffers the output and usually returns it when the command has finished executing.

Enscroll

Very nice. FYI: The stdout/stderr events callback argument 'data' is a buffer so call it with .toString() – Grass

For those of you who can't get spawn to work on Windows, have a look at this great answer. – Burgwell

exec is also an EventEmitter at least in latest. – Erdei

Also keep in mind that the callback will not be called, whenever the program outputs a newline. If you want to receive "events" from the child process, this process must flush the buffer (flush(stdout); in C) in order to fire events in Node.js. – Weatherbeaten

+1 on exec also being an EventEmitter.. spent 2 hours on refactoring my string into an args array (very long and complicated ffmpeg command line).. only to find out I didn't really need to. – Dwarf

It seems that when I run this in windows, the thing doesn't actually exit out and I have to hit CTRL+C. Edit: it seems the fix it to run process.exit(). – Honeybunch

The question is how do you solve the "progress bar" not being displayed as new line (for example webpack writes the progress into stderr). You can "process.stderr.write('\x1Bc')" every time to clear up the console above, but that's not a solution, because you clean all prior output – Alonaalone

It was helpful for me to find that process.stdout.write( data.toString() ); prevents adding a newline (like console.log does) after each stdout emission. – Aara

var spawn = require('child_process').spawn, git = spawn('git', ['clone', 'https://github.com/MaxySpark/Text-Clustering.git']); git.stdout.on('data', function (data) { console.log('stdout: ' + data.toString()); }); git.stderr.on('data', function (data) { console.log('stderr: ' + data.toString()); }); git.on('exit', function (code) { console.log('child process exited with code ' + code.toString()); }); its only print stderr: Cloning into 'Text-Clustering'... why not printing other output texts? and why is it on stderr? – Tribble

Use ls.stdout.setEncoding('utf8'); in order to not need to convert the buffer with data.toString() – Cesium

how to you retrieve the output though? the solution here only prints them in callbacks. What if the caller needs the stdout content for something else? – Jimmie

Use github.com/moxystudio/node-cross-spawn . It just works, has the same API, and gives you all of the output from the command. – Laine

How would we achieve the same in a world of promises? That is, if we run something like promisify(exec)('echo hello') or promisify(spawn)('echo world') – Alford

(Downvoted this answer, as it it wrong) - You can use either spawn or exec in the same way, they both return a ChildProcess object, the difference is that exec spawns a shell (such as bash) to execute the provided command, while spawn execute the command directly (with system execvp function on linux and mac os). You can use exec to output stdout while the command is being executed. – Barbarity

Answer 2

exec will also return a ChildProcess object that is an EventEmitter.

var exec = require('child_process').exec;
var coffeeProcess = exec('coffee -cw my_file.coffee');

coffeeProcess.stdout.on('data', function(data) {
    console.log(data); 
});

OR pipe the child process's stdout to the main stdout.

coffeeProcess.stdout.pipe(process.stdout);

OR inherit stdio using spawn

spawn('coffee -cw my_file.coffee', { stdio: 'inherit' });

Bunko

Looks like this can be simplified by just using pipe: coffeeProcess.stdout.pipe(process.stdout); – Overrun

@EricFreese's comment is what I was looking for, because I wanted to leverage stdout's characters replacement feature (harnessing protractor in a node script) – Vorfeld

Simpler: spawn(cmd, argv, { stdio: 'inherit' }). See nodejs.org/api/child_process.html#child_process_options_stdio for different examples. – Indopacific

+1 for @MorganTouvereyQuilling's suggestion to use spawn with stdio: 'inherit'. It produces more accurate output than exec and piping stdout/stderr, for example when displaying the progress information from a git clone. – Joline

Answer 3

There are already several answers however none of them mention the best (and easiest) way to do this, which is using spawn and the { stdio: 'inherit' } option. It seems to produce the most accurate output, for example when displaying the progress information from a git clone.

Simply do this:

var spawn = require('child_process').spawn;

spawn('coffee', ['-cw', 'my_file.coffee'], { stdio: 'inherit' });

Credit to @MorganTouvereyQuilling for pointing this out in this comment.

Joline

I found that when the subprocess uses formatted output like colored text, stdio: "inherit" preserves that formatting while child.stdout.pipe(process.stdout) does not. – Fritillary

This perfectly preserves output even on processes with complex output like the progress bars on npm installs. Awesome! – Mella

why this is not the accepted answer? it was the only one that worked for me and it's just 2 f* lines!!! – Whereat

This tip was helpful when executing some Symfony command-line applications that use progress bars. Cheers. – Premed

This should be the accepted answer–only thing that preserves perfect output representation and it's the simplest? yes please – Ammieammine

exactly what I was searching for, great answer, must be accepted – Tradesman

This should be the selected answer. – Autonomy

Answer 4

Inspired by Nathanael Smith's answer and Eric Freese's comment, it could be as simple as:

var exec = require('child_process').exec;
exec('coffee -cw my_file.coffee').stdout.pipe(process.stdout);

Christianize

This seems to work fine for simple commands like ls but fails for more complex commands such as npm install. I even tried piping both stdout and stderr to their respective process objects. – Bisayas

@Bisayas it may be because npm is writing in stderr (i saw some write the progress bar there). you can pipe also stderr, or extend Tongfa solution to listen on stderr. – Alonaalone

@Bisayas From what I've seen the most reliable way is spawn(command, args, { stdio: 'inherit' }), as suggested here #10232692 – Joline

Best answer, Thanks for this. Worked like a charm – Aegaeon

Answer 5

I'd just like to add that one small issue with outputting the buffer strings from a spawned process with console.log() is that it adds newlines, which can spread your spawned process output over additional lines. If you output stdout or stderr with process.stdout.write() instead of console.log(), then you'll get the console output from the spawned process 'as is'.

I saw that solution here: Node.js: printing to console without a trailing newline?

Hope that helps someone using the solution above (which is a great one for live output, even if it is from the documentation).

Lecythus

For even more accurate output use spawn(command, args, { stdio: 'inherit' }), as suggested by @MorganTouvereyQuilling here #10232692 – Joline

Answer 6

I have found it helpful to add a custom exec script to my utilities that do this.

utilities.js

const { exec } = require('child_process')

module.exports.exec = (command) => {
  const process = exec(command)
  
  process.stdout.on('data', (data) => {
    console.log('stdout: ' + data.toString())
  })
  
  process.stderr.on('data', (data) => {
    console.log('stderr: ' + data.toString())
  })
  
  process.on('exit', (code) => {
    console.log('child process exited with code ' + code.toString())
  })
}

app.js

const { exec } = require('./utilities.js')
    
exec('coffee -cw my_file.coffee')

Placencia

It is working properly but its format is not the same as it comes while installing npm like the progress bar, and color text. – Terriss

Answer 7

After reviewing all the other answers, I ended up with this:

function oldSchoolMakeBuild(cb) {
    var makeProcess = exec('make -C ./oldSchoolMakeBuild',
         function (error, stdout, stderr) {
             stderr && console.error(stderr);
             cb(error);
        });
    makeProcess.stdout.on('data', function(data) {
        process.stdout.write('oldSchoolMakeBuild: '+ data);
    });
}

Sometimes data will be multiple lines, so the oldSchoolMakeBuild header will appear once for multiple lines. But this didn't bother me enough to change it.

Adlare

Answer 8

child_process.spawn returns an object with stdout and stderr streams. You can tap on the stdout stream to read data that the child process sends back to Node. stdout being a stream has the "data", "end", and other events that streams have. spawn is best used to when you want the child process to return a large amount of data to Node - image processing, reading binary data etc.

so you can solve your problem using child_process.spawn as used below.

var spawn = require('child_process').spawn,
ls = spawn('coffee -cw my_file.coffee');

ls.stdout.on('data', function (data) {
  console.log('stdout: ' + data.toString());
});

ls.stderr.on('data', function (data) {
  console.log('stderr: ' + data.toString());
});

ls.on('exit', function (code) {
  console.log('code ' + code.toString());
});

Audreyaudri

Answer 9

Here is an async helper function written in typescript that seems to do the trick for me. I guess this will not work for long-lived processes but still might be handy for someone?

import * as child_process from "child_process";

private async spawn(command: string, args: string[]): Promise<{code: number | null, result: string}> {
    return new Promise((resolve, reject) => {
        const spawn = child_process.spawn(command, args)
        let result: string
        spawn.stdout.on('data', (data: any) => {
            if (result) {
                reject(Error('Helper function does not work for long lived proccess'))
            }
            result = data.toString()
        })
        spawn.stderr.on('data', (error: any) => {
            reject(Error(error.toString()))
        })
        spawn.on('exit', code => {
            resolve({code, result})
        })
    })
}

Succubus

Answer 10

I found another way to do it:

const { spawn } = require('child_process');

export const execCommand = async (command) => {
  return new Promise((resolve, reject) => {
    const [cmd, ...args] = command.split(' ');
    const childProcess = spawn(cmd, args);
    childProcess.stdout.on('data', (data) => {
      process.stdout.write(data.toString());
    });
    childProcess.stderr.on('data', (data) => {
      process.stderr.write(data.toString());
    });
    childProcess.on('error', (error) => {
      reject(error);
    });
    childProcess.on('exit', (code) => {
      if (code === 0) {
        resolve();
      } else {
        reject(new Error(`Command exited with code ${code}.`));
      }
    });
  });
};

This code gives and ability to get real-time output from executed command and redirect all stdout and stderr to parent process. It also allows using the command the same way you use it in bash/sh (single string input). Here I use process.stdout.write for more accurate output instead of console.log that is used in other answers.

Usage:

await execCommand('sudo apt-get update');
await execCommand('sudo apt-get install -y docker.io docker-compose');

Note: compared to exec it does not support execution of multiple commands using &&. So each single command should be executed with a single execCommand statement.

And here is a simplified version that supports both realtime streaming and shell execution:

const { spawn } = require('child_process');

export const execCommand = async (command) => {
  return new Promise((resolve, reject) => {
    const childProcess = spawn(command, { 
      stdio: 'inherit',
      shell: true
    });
    childProcess.on('error', (error) => {
      reject(error);
    });
    childProcess.on('exit', (code) => {
      if (code === 0) {
        resolve();
      } else {
        reject(new Error(`Command exited with code ${code}.`));
      }
    });
  });
};

Usage:

await execCommand('sudo apt-get update && sudo apt-get install -y docker.io docker-compose');

Karinekariotta