Stream stdout from child process to browser via expressjs

Question

We have an app built using nodejs, express and child_process.spawn. One requirement is that we need to spawn a process at runtime and capture it's output and present it to the user.

We have that working. However, we need to figure out a way to stream the output instead of waiting until the child process exists.

We've looked around and couldn't find any clear cut examples and were wondering if anyone here had any ideas?

Everything is working. We just don't care for the user experience of having to wait for the command to finish before showing the entire output. If we could stream it, that would be ideal so as the stdout data event was triggered, the browser would update with the new data. As right now it does come in chunks as opposed to one big blob.. so it's suited quite nicely to do this.

Haversine

Answer 1

The response object of an Express route is also an instance of writable stream, so that allows you to pipe the child process' stdio streams to the response.

app.get('/path', function(req, res) {
  var child = spawn('ls', ['-al']);
  child.stdout.pipe(res);
});

Fustigate

It's still waiting until the process finishes before it starts showing the content in the browser. It was definitely a lot easier to do it your way though, so thanks for the tip. However, it's still waiting until the process is done. – Haversine

This does not wait until the process is done. If it seems like so, that means that the process isn't writing anything to its stdout stream until it finishes, which would be the process' problem, not Node.js'. It is streaming in this case. – Fustigate

That's odd as if I bind to the child.stdout I receive data at intermittent times as expected. For now, we've implemented a socket.io approach that seems to be working. It was more effect then we'd like, but it will do. – Haversine

Answer 2

Another approach would be to use Socket.IO. Socket.IO is designed for this kind of real-time event-based communication between the client and the server.

The key is to flush the standard output stream after writing to it. This should trigger the "childprocess.stdout.on('data', ...)" event.

How this is done depends on what language your child process is written in. Say you're running a python script and writing to the standard output using "print", then you'd need to do this:

import sys

# Flush after each print to stdout
sys.stdout.flush()

Here's a simple example that implements what you're asking using Socket.IO. Server side code:

var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var child = require('child_process');

app.get('/', function(req, res){
  res.sendFile(__dirname + '/index.html');
});

http.listen(8000, function(){
	console.log('listening on *:8000');
});

io.on('connection', function(socket){
	console.log('new user connected');
	
	var python = child.spawn( 'python', ['compute.py'],[]);

	var chunk = '';
	python.stdout.on('data', function(data){
		chunk += data
		socket.emit('newdata', chunk);
	} );
	
	python.stderr.on('data', function (data) {
		console.log('Failed to start child process.');
	})
})

On the client, you could then register an event listner:

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io();
   
  socket.on('newdata', function(d){
     // Do something with the data
  })
</script>

Inextensible