Node pipe to stdout -- how do I tell if drained?

Question

The standard advice on determine whether you need to wait for drain event on process.stdout is to check whether it returns false when you write to it.

How should I check if I've piped another stream to it? It would seem that that stream can emit finish before all the output is actually written. Can I do something like?

upstreamOfStdout.on('finish', function(){
  if(!process.stdout.write('')) {
    process.stdout.on('drain', function() { done("I'm done"); });
  }
  else {
    done("I'm done"); 
  }
});
upstreamOfStdout.pipe(process.stdout);

I prefer an answer that doesn't depend on the internals of any streams. Just given that the streams conform to the node stream interface, what is the canonical way to do this?

EDIT:

The larger context is a wrapper:

new Promise(function(resolve, reject){
  stream.on(<some-event>, resolve);
  ... (perhaps something else here?)
});

where stream can be process.stdout or something else, which has another through stream piped into it.

My program exits whenever resolve is called -- I presume the Promise code keeps the program alive until all promises have been resolved.

I have encountered this situation several times, and have always used hacks to solve the problem (e.g. there are several private members in process.stdout that are useful.) But I really would like to solve this once and for all (or learn that it is a bug, so I can track the issue and fix my hacks when its resolved, at least): how do I tell when a stream downstream of another is finished processing its input?

Leggett

Answer 1

Instead of writing directly to process.stdout, create a custom writable (shown below) which writes to stdout as a side effect.

const { Writable } = require('stream');
function writeStdoutAndFinish(){
  return new Writable({
    write(chunk, encoding, callback) {
      process.stdout.write(chunk,callback);
    },
  });
};

The result of writeStdoutAndFinish() will emit a finish event.

async function main(){
  ...
  await new Promise((resolve)=>{
    someReadableStream.pipe(writeStdoutAndFinish()).on('finish',()=>{
      console.log('finish received');
      resolve();
    }) 
  });
  ...
}

In practice, I don't that the above approach differs in behavior from

async function main(){
  ...
  await new Promise((resolve)=>{
    (someReadableStream.on('end',()=>{
      console.log('end received');
      resolve();
    })).pipe(process.stdout)
  });
  ...
}

Kidnap

Grumble... ok :) ... still there should be a better way of doing this "in principle" even if work-arounds work without problem. – Leggett

Just want to say I searched for hours before finally coming across this answer. Thank you so much you saved my day (and my hair). – Irrespirable

Fantastic workaround, noting that the "in practice, ..." solution posted at the bottom of the answer does NOT work. Also I can add an addendum to anyone wanting to use this: Create your someReadableStream as a Stream.PassThrough and write() then end() it. – Freshman

Answer 2

First of all, as far as I can see from the documentation, that stream never emits the finish event, so unlikely you can rely on that.

Moreover, from the documentation above mentioned, the drain event seems to be used to notify the user about when the stream is ready to accept more data once the .write method returned false. In any case you can deduce that that means that all the other data have been written. From the documentation for the write method indeed we deduce that the false value (aka please stop pushing data) is not mandatory and you can freely ignore it, but subsequent data will be probably stuffed in memory letting the use of it to grow up.

Because of that, basing my assumption on the sole documentation, I guess you can rely on the drain event to know when all the data have been nicely handled or are likely to be flushed out.

That said, it looks to me also that there is not a clear way to definitely know when all the data have been effectively sent to the console.

Finally, you can listen the end event of the piped stream to know when it has been fully consumed, no matter if it has been written to the console or the data are still buffered within the console stream. Of course, you can also freely ignore the problem, for a fully consumed stream should be nicely handled by node.js, thus discarded and you have not to deal with it anymore once you have piped it to the second stream.

Asterisk

The problem I am having is that without listening for drain the program terminates before everything is written, but if I listen for drain it sometimes never occurs. I presume drain is emitted only when data is buffered, but I don't know whether data is buffered or not. Their seems to be no documented way to ask whether a stream has finished writing or not. – Leggett

The buffering takes place when the write method returns false, or at least that's what the documentation states. Thus the drain event is subsequent to such a case. Keep in mind that your program will stay alive as far as you have something meaningful on the loop to be executed. Have you tried listening on the first stream for the end event, not the final one? – Asterisk

I believe that end is deprecated for writable streams -- finish is the event to listen for. As you can see from my example code, I do listen to finish. You are right about "buffering takes place when the write method returns false" -- the current question is what to do when you can't see write return because you have used pipe to send the data. Eg is writing a null string after receiving finish from upstream officially sanctioned? I would greatly appreciate a response from a maintainer, or a reference to something they have written, as doc seems inconclusive. – Leggett

The end event is not deprecated and be aware that finish is never emitted on stdout (accordingly to the documentation). – Asterisk

hmmm... end is no longer in the documentation for a writable stream at least?. Nothing seems to be added in the process.stdout docs. (?) (Also note -- see sample code -- I listen for finish on the upstream stream.) Do you have a reference somewhere that says end is not deprecated for writable streams? – Leggett

well, you are right, I was referring to an older documentation, sorry. it looks like the end event is no longer there. Anyway, listening for the finish event on the upstream does not give you any guarantee. as an example, if the upstream flush all the data and the stream after that one is buffering, the former will still emit the finish event for it is not a problem of theirs you having pending data somewhere. – Asterisk

yes... so my idea was to listen to the upstream finish which should at least tell me that the data was sent (even if buffered), then to write an empty string to check if it was buffered. This code even "works" (in a few tests)... the question is whether its working by accident or whether it really has to work. For instance is writing an empty string guaranteed to tell me whether the previous data has been flushed (or if not, is there is another way...)? – Leggett