how to turn Child_process.spawn's "Promise" syntax to "async/await" syntax

Question

So I have this code and I'm trying to understand the async/await syntax in full depth. The below is the Promise version of the code:

function callToolsPromise(req) {
    return new Promise((resolve, reject) => {
        let pipshell = 'pipenv';
        let args = ['run', 'tools'];
        req.forEach(arg => {
            args.push(arg)
        });
        tool = spawn(pipshell, args);
        tool.on('exit', (code) => {
            if (code !== 0) {
                tool.stderr.on('data', (data) => {
                    reject(data);
                });
            } else {
                tool.stdout.on ('data', (data) => {
                    resolve(JSON.parse(data)):
                });
            }
        });
    })
}

I have some python code I want to execute in tools/__main__.py so that's why I'm calling "pipenv".

Here's my attempt to do write it in async/await way (which actually works):

async function callToolsAsync(req) {
    let pipshell = 'pipenv';
    let args = ['run', 'tools'];
    req.forEach(arg => {
        args.push(arg)
    });
    let tool = spawn(pipshell, args);
    for await (const data of tool.stdout) {
        return data
    }
}

But all I did was copy and paste from someone's example where I have for await... loop.

Therefore I've been trying to rewrite this same code so that I can actually understand it but I've been failing for days now.

Are there any other ways to write this code with async/await way without using the for await... loop?

Also I have no idea how I can access data except for using the .then syntax:

callToolsAsync(['GET','mailuser'])
.then(console.log)

How else would I access "data" from resolve(data)?

Many thanks.

Corporator

Waiting for data on stdout/stderr only after the exit event seems like a very bad idea – Inane

Answer 1

There are probably not better ways to write the code using async/await than using the for async (chunk of stream) syntax in Node. Node streams implement an asynchronous iterator specifically to allow doing so.

This article on 2ality has more in-depth explanation and discussion. MDN articles on Symbol.asyncIterator and for-await...of cover asynchronous iteration more generally.

Once using asychronous iteration has been decided it must be used in an async function, which will return a promise.

While using a then clause on the returned promise is a completely normal way of getting the data, you could also await the result of callToolsAsync(req) - provided of course that the call is coded inside an async function so that await is in a valid context.

---

The following code experiment gets the stdio and stderr output, and the exit code from a child process. It doesn't use Python or parse data.

main.js (type node main.jsto run)

// main.js
async function spawnChild() {
    const { spawn } = require('child_process');
    const child = spawn('node', ["child.js"]);

    let data = "";
    for await (const chunk of child.stdout) {
        console.log('stdout chunk: '+chunk);
        data += chunk;
    }
    let error = "";
    for await (const chunk of child.stderr) {
        console.error('stderr chunk: '+chunk);
        error += chunk;
    }
    const exitCode = await new Promise( (resolve, reject) => {
        child.on('close', resolve);
    });

    if( exitCode) {
        throw new Error( `subprocess error exit ${exitCode}, ${error}`);
    }
    return data;
}

spawnChild().then(
    data=> {console.log("async result:\n" + data);},
    err=>  {console.error("async error:\n" + err);}
);

child.js

// child.js
console.log( "child.js started"); //  stdout
setTimeout( finish, 1000);
function finish() {
    console.log( "child.js: finish() call");  //  stdout 
    console.error("child exit using code 1"); //  stderr
    process.exit(1);
}

This showed

• A console warning that async iteration of a readable stream is still experimental in node,
• The for await (chunk of stream) loops seem to loop until the stream is closed - in this case meaning that await will wait on an open stream that doesn't have data available at the time.
• retrieving stdout and stderr content from their pipes, and getting the exit code can be done in no particular order since retrieval is asynchronous.
• Amalgamating chunks of data arriving through pipes from another process is not optional - console logs from the child process came through separately.

Outdated

Would you be able to provide a example of "monitoring the process exit codes or data appearing on stderr"? – Corporator

Thank you for the amazing explanation. This has clarified a lot of questions I had for example the behavior of running multiple for await loops. Thank you for the clarification. – Corporator

This "for await (const chunk of child.stdout)" won't block, then? Meaning I will get stdout and stderr simultaneously? – Hessian

@Hessian An await ... of construct won't exit the loop until the stream being read is closed or a break statement is executed within the loop body, blocking monitoring two streams simulataneously. It doesn't block the child process writing to stdio output - data written is stored in the stream's pipe until read. If stderr and sdout need to be monitored and action taken before the child process finishes, then alternating between reading streams with asyncrhonous iterators is likely to be more complex than other solutions. – Outdated

Answer 2

Its important to understand that async/await and promises are the same, just different syntax.

So every async function returns a promise!

so assuming you have a function returining a promise:

function foo() {
  return new Promise(resolve => setTimeout(() => resolve("done"), 1000));
}

there are two ways to consume the value.

Promise style:

function test() {
  foo().then(value => console.log(value));
}

or async await:

async function test() {
  const value = await foo();
  console.log(value);
}

Now its important to understand that your original callToolsPromise function is not promise style. when working with promises you never call new Promise. Basically the entire idea of of new Promise is to convert asynchronous non-promise code (and so non-async/await as its the same) to Promises (and so async/await).

Now asynchronous does not mean async/await but a more general concept. The other common way to handle asynchronity are callbacks.

So tool.on('exit', (code) => { is asynchronous, but neither a Promise nor async/await.

So the wrapping new Promise is basically used to convert this to a Promise style function that can be used as Promise or with async/await.

---

a last word about this snippet:

for await (const data of tool.stdout) {
    return data
}

this is a bit problematic. While node streams are async generators, this will return after the first chunk and break as soon as the stream receives multiple chunks. So you either should replace return by yield and return an async generator, or concatenate the buffers with the loop and return after the loop completes.

Bathelda

@Bathelda I'm thankful for the insightful explanation. But I don't like the attitude of "because I don't know it, it's invalid". Rather assuming what the code would return, I suggest testing it first... – Corporator

Thanks, thats why my AFAIK was there. However you definitly need to merge the chunks. return inside the loop will only work when there is only 1 chunk. – Bathelda

You're right, My apologies. Thank you for the suggestion. Fixed my code :) – Corporator

The more and more I come back to this, It's made me realize the important concepts in this answer. Thank you @Lux. Have a great day. – Corporator

Answer 3

direct answer:

const { spawn } = require("child_process");
const process = require("process");

function cmd(...command) {
  let p = spawn(command[0], command.slice(1));
  return new Promise((resolve) => {
    p.stdout.on("data", (x) => {
      process.stdout.write(x.toString());
    });
    p.stderr.on("data", (x) => {
      process.stderr.write(x.toString());
    });
    p.on("exit", (code) => {
      resolve(code);
    });
  });
}

async function main() {
  await cmd("bash", "-c", "echo one; sleep 1; echo two; sleep 1");
  console.log("This must happen last.");
}

main();
// run main() again & again , it will run in-parallel with the first main()

Constringent

Answer 4

I solved this problem by using await-spawn

https://www.npmjs.com/package/await-spawn

child_process.spawn() wrapped in a Promise for doing async/await.

Trumpetweed

Thanks. a very elegant and easy-to-use solution. fits perfectly from my use case. – Bondstone

Thank yooooooooouuuuuuu! A very underrated lib! IMHO the best way to handle errors in an interactive context (stdio: inherit) – Taker