Result of ps au | grep ssh different in Node.js (using spawn/pipe) vs shell

Question

I'm playing around with node streams and child processes. So I want to emulate next shell command with pipes:

ps au | grep ssh

So I wrote next code:

var spawn = require('child_process').spawn;
var ps    = spawn('ps',   ['au']);
var grep  = spawn('grep', ['ssh']);

ps.stdout.pipe(grep.stdin);

grep.stdout.on('data', function(data) { console.log(data) });

Then I run it, but nothing happens. What did I do wrong?

P.S. - I know about:

require('child_process')
   .exec('ps au | grep ssh', function(err, stdout, stderr) { 
       ... 
   }). 

I'm just playing around with Node.js, and I want to understand what's wrong with this example.

Update 1:
It appeared that with grep bash program works as expected, but with grep ssh there is no result. Although ps au | grep ssh gives me this result:

vagrant 11681 0.0 0.1 10464 916 pts/0 S+ 07:54 0:00 grep --color=auto ssh.

Exeter

Your code works for me (except with bash instead of ssh). If you do ps au | grep ssh at your shell prompt, do you see output? – Nubbly

@Nubbly Hmm, interesting. With bash command it works (result is the same as ps au | grep bash). With ssh there is no result. Although ps au | grep ssh give me result: vagrant 11681 0.0 0.1 10464 916 pts/0 S+ 07:54 0:00 grep --color=auto ssh. – Exeter

Which userid are these nodejs scripts running under? Nobody or web user perhaps, instead of your login user? – Branch

I think there is some kind of timing issue because I can see the output sometimes but not always. – Capacitance

@Branch I call this script from console: node pipe_example.js using vagrant (virtual machine with OS Linux Ubuntu 14.04). So user - vagrant, user id - 1000. – Exeter

@DonghoYoo Yes, I'm also thinking so. But I don't understand why and where? pipe command must push new data to grep.stdin, so we must have data on grep.stdout. – Exeter

@DonghoYoo I tried wrap ps.stdout.pipe(grep.stdin) into process.nextTick. After this program begin works somtimes, but not always (as you already said) – Exeter

No sure why you are surprised, vagrant 11681 0.0 0.1 10464 916 pts/0 S+ 07:54 0:00 grep --color=auto ssh. is the grep ssh process. If you do ps au | grep ssh it is up shell if it starts grep before ps au is launched, but this is not necessary as it also could buffer the result of ps au while starting grep, so whether you'll see the grep ssh of the pipe in the listing of ps au depends on the behavior of the used shell and the scheduling of the OS. The sample of Dongho Yoo might randomly not display grep in the process listing, but that it very unlikely – Concoff

@Concoff As documentation for streams[nodejs.org/api/stream.html#stream_class_stream_readable] says there are two mode flowing and non-flowing. In non-flowing mode all data is buffered, in flowing mode data is processed (emitting data event). You can switch to 'flowing' mode only by adding listener to data event, or by calling stream.resume() method, or by calling pipe() method. So there must be no differences which command (ps or grep) run first. Data from ps command must be buffered and passed to grep command after pipe(grep.stdin) call. – Exeter

@Concoff Maybe I wrong. Feel free to correct me. – Exeter

Answer 1

When you call ps it will list all currently running processes matching the passed options. Which might look for ps au something like this:

tniese  3251   0,0  0,0  2479028   3004 s000  S+    4:06am   0:00.03 -bash
root    4453   0,0  0,0  2452408    876 s004  R+    4:06pm   0:00.00 ps au

When you call ps au | grep ssh in the shell grep will filter that result to only show the lines containing ssh.

If the grep is launched by the shell before ps creates its listing then the output before the filtering would be:

tniese  3251   0,0  0,0  2479028   3004 s000  S+    4:06am   0:00.03 -bash
root    4453   0,0  0,0  2452408    876 s004  R+    4:06pm   0:00.00 ps au
tniese  4478   0,0  0,0  2441988    596 s000  R+    4:06pm   0:00.00 grep ssh

The grep process will match its own entry as it contains the passed parameters, so the filtered result would be:

tniese  4478   0,0  0,0  2441988    596 s000  R+    4:06pm   0:00.00 grep ssh

Lets look what is happening with your code:

var spawn = require('child_process').spawn;
var ps    = spawn('ps',   ['au']);
var grep  = spawn('grep', ['ssh']);

ps.stdout.pipe(grep.stdin);

With spawn you tell the OS to start the process ps, ps does not need to wait to run until the output could be piped to anyplace but could start before that, it might only be forced to wait when it tries to write to the its output stream. Then your spawn grep, but at the time grep is launched ps might already created the process listing internally and cause of that it does not contain the grep process. The output of ps is then passed to grep. But as this output is missing grep ssh it won't show that line.

Wether grep will appear in your listing or not is highly OS dependent. Generally you should assume that it is random if it is listed or not. Or you would need to wait until ps exits and launch grep after that.

You need to always keep in mind that current OS have preemptive multitasking and that the scheduler might pause node right after spawn('ps', ['au']); and continue the process right after ps created/requested the listing.

I hope this explanation is a bit clearer then my comment.

Concoff

Year. I've got your point. I think nobody can explain better, so mark you answer as accepted. But one last question: don't you think that node must buffer result data for ps command in stdout stream? Or at least have option to do so? – Exeter

Create the issue for node. Let's core team decides correctness of such behavior. – Exeter

@Exeter I assume that node or the OS itself already do buffering for the stdout of the spawned process, but that does not really matte for your described problem. As long as ps is spawnd before grep, it is more likely that grep won't appear in the listing then that it would appear, but this depends on the scheduler of the system what would happen. For the command ps au | grep ssh shell will most likely spawn grep before it spawns ps, and that's why you see grep as result in your shell. It is nothing wrong with both behaviors. – Concoff

Answer 2

I've spawned grep before ps and now it works well. I think it must be a timing issue. Try this.

var spawn = require('child_process').spawn;
var grep  = spawn('grep', ['ssh']);
var ps    = spawn('ps',   ['au']);

ps.stdout.pipe(grep.stdin);

grep.stdout.on('data', function(data) { 
  console.log(data.toString("utf8")); 
});

Capacitance

Year, it works! But I don't understand why? For me this is absolutely illogical. I'll wait a little - maybe someone knows something about it, maybe it's a feature, not a bug. If not - mark your answer as accepted and create issue in node github repository – Exeter