Child_process handling a STDOUT stream with carriage return (\r)

Question

I'm writing a simple(ish) application that allows an internal system at work to request a copy process (using rsync) from a remote server to another remote server instigated using a REST call.

I'm already familiar enough with the express framework, and have just started experimenting with the child_process library, and stumbled upon a small problem.

I am successfully starting rsync processes using node's childProcess.spawn(), my issue is that rsync outputs its progress line-buffered with a carriage return (\r) instead of an newline (\n). As a result the STDOUT event, process.stdout.on('data', {}) is only called once before saying it's setting up the transfer, and then after the copy has finished, as STDOUT data isn't flushed on a carriage return, as the progress updates, only a newline, when the job finishes.

There is a switch in the latest version of rsync (3.1.0) to change the output buffer endings to \n instead of \r but unfortunately, the company I work at will not be adopting this version for a long time.

I'm spawning and reading the child_process in the usual way....

var doCopy = function (onFinish) {
    var process = childProcess.spawn('ssh', [
        source.user + "@" + source.host,
        "rsync",
        "-avz",
        "--progress",
        source.path + source.file,
        "-e ssh",
        dest.user + "@" + dest.host + ":" + dest.path
    ]);

    process.on('error', function (error) {
        console.log("ERR: " + error.code);
    })

    process.stdout.on('data', function (data) {
        console.log("OUT: " + data);
    });

    process.stderr.on('data', function (data) {
        console.log("ERR: " + data);
    });

    process.on('close', function (code) {
        console.log("FIN: " + code);
        if(onFinish){
            onFinish(code);
        }
    });
}

..and the console output is....

OUT: building file list ... 
OUT: 
1 file to consider

OUT: test4.mp4

         32,768   0%    0.00kB/s    0:00:00  
    169,738,240  32%  161.84MB/s    0:00:02  
    338,165,760  64%  161.32MB/s    0:00:01  
    504,692,736  96%  160.53MB/s    0:00:00  
    524,288,000 100%  160.35MB/s    0:00:03 (xfr#1, to-chk=0/1)

OUT: 
sent 509,959 bytes  received 46 bytes  113,334.44 bytes/sec
total size is 524,288,000  speedup is 1,028.01

FIN: 0

So you can see that the stdout.on('data,) is only called when rsync outputs a new line (where there is an 'OUT:').

My question is, can I change this? Maybe put the stream through a transform to flush when a \r occurs? Then I can regex the line and offer a progress update back again.

Failing this, I suppose my only other option would be to spawn another process to monitor the growing file?

Any help/advise is very much appreciated.

Skein

Answer 1

I've found this very good module that does exactly what I was trying to achieve. Allows me to delimit the stdout buffer by any character (in my case '\r') and trigger a new stdout event to handle the data. Like....

var splitter = process.stdout.pipe(StreamSplitter("\r"));

splitter.on('token', function (data) {
    console.log("OUT: " + data);
});

splitter.on('done', function (data) {
    console.log("DONE: " + data);
});

Skein

Answer 2

I have made a custom stream Transform subclass for this purpose. The pro is that you can also tag your lines with your child process favorite name and it handles given \r the right way.

Feel free to try my implementation: (in typescript but you could easily remove all type stuff)

import { ChildProcessWithoutNullStreams } from "child_process";
import { Transform } from "stream";

export default class LineTagTransform extends Transform {
  lastLineData = '';
  tag = '';

  constructor(tag?: string) {
    super({ objectMode: true });

    this.tag = tag || '';
    if (tag && !tag.endsWith(' ')) this.tag += ' ';
  }

  _transform(chunk: Buffer | string | any, encoding: string, callback: Function) {
    let data: string = chunk.toString().replace(/\r(?!\n)/, '\n\r');
    if (this.lastLineData) data = this.lastLineData + data;

    let lines = data.split(/\r?\n/);
    this.lastLineData = lines.splice(lines.length - 1, 1)[0];

    for (const line of lines) {
      if (line.startsWith('\r')) {
        this.push(`\r${this.tag}${line.substring(1)}`);
      } else {
        this.push(`\n${this.tag}${line}`)
      }
    }
    callback();
  }

  _flush(callback: Function) {
    if (this.lastLineData) {
      if (this.lastLineData.startsWith('\r')) {
        this.push(`\r${this.tag}${this.lastLineData.substring(1)}`);
      } else {
        this.push(`\n${this.tag}${this.lastLineData}`)
      }
    }
    this.lastLineData = '';
    callback();
  }

  static wrapStreams(child: ChildProcessWithoutNullStreams, tag?: string, stdout: NodeJS.WriteStream = process.stdout, stderr: NodeJS.WriteStream = process.stderr) {
    child.stdout.pipe(new LineTagTransform(tag)).pipe(stdout);
    child.stderr.pipe(new LineTagTransform(tag)).pipe(stderr);
  }
}

Then the easiest way to use it:

const child = spawn('./DownloadUnicorn.exe', options);
LineTagTransform.wrapStreams(child, '[unicorn]');

Outputs:

[unicorn] Start DownloadUnicorn!
[unicorn] Downloading [=====-----] 50%
...

With download bar animation on a single line! \o/

HTH! ;)

Regardant