NodeJS exec/spawn stdout cuts off the stream at 8192 characters

Question

I'm on MacOS, Node v12 and using a child process (exec/spawn/execSync/spawnSync) to execute a shell command which can return more than 8192 characters. However, back in my nodeJS method that invokes the child process, I only ever get up to 8192 characters and no more than that. (8192 seems to be the default pool size for a buffer).

I've tried increasing the maxBuffer size in the Options to anything larger than 8192 but it does not affect anything.

I've also tried running the same command with exec, spawn, execSync and spawnSync and they all behave the same way. Same result.

When I run:

 exec(shellCommand, { encoding: "buffer", maxBuffer: 16384 }, (error, stdout, stderr) => {
     console.log('stdout--> length: ', stdout.length, '<--->', stdout)
 });

I get:

stdout--> length: 8192 <---> <Buffer 7b 22 72 65 73 75 6c 74 22 3a 5b 7b 22 70 72 6f 6a 65 63 74 4e 61 6d 65 22 3a 22 73 65 65 64 73 22 2c 22 74 65 6d 70 6c 61 74 65 4e 61 6d 65 22 3a 22 ... 8142 more bytes>

I know that the data coming back is larger than 8192 because when I run the shell command in a shell and check the length it is greater than 8192.

Also, and this is the puzzling bit, when I set the child process' stdio option to 'inherit' such as:

execSync(shellCommand, { encoding: "buffer", stdio:"inherit" });

(which says to use the parents stdout, in my case that is the NodeJS' console)

I see the full response back in the console where NodeJS is running.

I have also read a similar issue on github but it hasn't really helped.

How do I go about executing a shell command in NodeJS and getting the full response back?

Forjudge

Answer 1

Turns out that the shell command had a process.exit() statement that is being called before the stdout buffer is fully flushed.

So stdout will send 8192 chars and since it's asynchronous the process will go on to the next statement, one of them being process.exit() and it will kill the process before flushing out the rest of the stdout buffer.

TL;DR - exec/spawn works correctly, shell command exits before stdout fully flushed

Forjudge

Answer 2

try this :

const { spawn } = require('child_process');
const cmd = spawn('command', ['arg1', 'arg2']);
let bufferArray= []
/*cmd.stdout.setEncoding('utf8'); sets encdoing
 defualt encoding is buffer
*/
cmd.stdout.on('data', (data) => {
  console.log(`stdout: ${data}`);
  bufferArray.push(data)
});

cmd.stderr.on('data', (data) => {
  console.error(`stderr: ${data}`);
});

cmd.on('close', (code) => {
  console.log(`child process exited with code ${code}`);
  let dataBuffer =  Buffer.concate(bufferArray];
  console.log(dataBuffer.toString())
});

this could be useful: Node.js spawn child process and get terminal output live

Scandic

Answer 3

The default buffer size for child_process.exec is 1MB, so try not passing a maxBuffer; however, it would be much better to use child_process.spawn so that you get the output as a stream.

Votaw

tried w/o the maxBuffer and with spawn but getting the same result – Forjudge