nodejs/express - stream stdout instantly to the client

Asked 21/2, 2014 at 12:43 Answered 23/2, 2014 at 21:11

Solved node.js express spawn child-process

I spawned the following child: var spw = spawn('ping', ['-n','10', '127.0.0.1']) and I would like to receive the ping results on the client side (browser) one by one, not as a whole.

So far I tried this:

app.get('/path', function(req, res) {
   ...
   spw.stdout.on('data', function (data) {
      var str = data.toString();
      res.write(str + "\n");
   });
   ...
}

and that:

...
spw.stdout.pipe(res);
...

In both cases browser waits 10 of the pings to complete, and then prints the result as a whole. I would like to have them one by one, how to accomplish that?

(Client is just making a call to .../path and console.logs the result)

EDIT: Although I do believe that websockets are necessary to implement this, I just want to know whether there are any other ways. I saw several confusing SO answers, and blog posts (in this post, at step one OP streams the logs to the browser) which didn't help, therefore I decided to go for a bounty for some attention.

Incipit answered 21/2, 2014 at 12:43 Comment(0)

Here's a complete example using SSE (Server sent events). This works in Firefox and probably Chrome too:

var cp = require("child_process"),
         express = require("express"),
         app = express();

app.configure(function(){
    app.use(express.static(__dirname));
});


app.get('/msg', function(req, res){
    res.writeHead(200, { "Content-Type": "text/event-stream",
                         "Cache-control": "no-cache" });

    var spw = cp.spawn('ping', ['-c', '100', '127.0.0.1']),
    str = "";

    spw.stdout.on('data', function (data) {
        str += data.toString();

        // just so we can see the server is doing something
        console.log("data");

        // Flush out line by line.
        var lines = str.split("\n");
        for(var i in lines) {
            if(i == lines.length - 1) {
                str = lines[i];
            } else{
                // Note: The double-newline is *required*
                res.write('data: ' + lines[i] + "\n\n");
            }
        }
    });

    spw.on('close', function (code) {
        res.end(str);
    });

    spw.stderr.on('data', function (data) {
        res.end('stderr: ' + data);
    });
});

app.listen(4000);

And the client HTML:

<!DOCTYPE Html>
<html> 
<body>
   <ul id="eventlist"> </ul>

   <script>              
    var eventList = document.getElementById("eventlist");
    var evtSource = new EventSource("http://localhost:4000/msg");

    var newElement = document.createElement("li");
    newElement.innerHTML = "Messages:";
    eventList.appendChild(newElement);


    evtSource.onmessage = function(e) {
        console.log("received event");
        console.log(e);
        var newElement = document.createElement("li");

        newElement.innerHTML = "message: " + e.data;
        eventList.appendChild(newElement);
    };      

    evtSource.onerror = function(e) {
        console.log("EventSource failed.");
    };

    console.log(evtSource);

    </script>

</body>
</html>

Run node index.js and point your browser at http://localhost:4000/client.html. Note that I had to use the "-c" option rather than "-n" since I'm running OS X.

Otherness answered 23/2, 2014 at 21:11 Comment(7)

exactly what I am looking for, will give your bounty once it allows me :) imgur.com/oLLbGXU – Incipit 24/2, 2014 at 11:36

what's a good solution to cancel the event stream when the user navigate from the page? – Pappose 10/4, 2017 at 20:18

@AhmedAhmed I would assume that if the user had closed the webpage either res.write() would throw an exception or the 'close' event would be triggered. Why don't you try it yourself and report your results here? – Otherness 11/4, 2017 at 6:40

I did try it actually :) from the client you need to call evtSource.close(); and from the server you need to listen to the close event req.connection.addListener("close", function () { console.log('client closed'); spw.stdin.pause(); spw.kill(); }); – Pappose 11/4, 2017 at 14:35

7 years later it still works 👍, but app.configured should be removed (just app.use(express.static(__dirname)); is enough) – Himeji 6/4, 2021 at 13:0

Great answer. Just two notes: 1) app.configure was removed from express v4 (see Ahmeds comment above). 2) The request was, for whatever reason, blocked by CORS (which was solved simply by installing cors module and adding app.use(cors()) line. – Carloscarlota 10/1, 2022 at 6:59

This has worked well for me, with with one issue: The calls to res.end are being received by my browsers as an error event, which in turn is causing the EventSource request to start over unless explicitly handled and closed. – Sham 13/4, 2022 at 15:58

If you are using Google Chrome, changing the content-type to "text/event-stream" does what your looking for.

res.writeHead(200, { "Content-Type": "text/event-stream" });

See my gist for complete example: https://gist.github.com/sfarthin/9139500

Clint answered 21/2, 2014 at 18:1 Comment(3)

Can you try your gist with var spw = cp.spawn('ping', ['127.0.0.1', '-n', '10']), and confirm that you can see real time console on the browser as well? Upon @jibsales answer I started to think that real time browser is only possible with sockets, but your answer confused me... – Incipit 22/2, 2014 at 23:47

just tried it, does not work. if you send 10 pings, your solution waits till all complete, then chrome displays all... – Incipit 23/2, 2014 at 9:51

@anvarik: See the code in my answer. It's based on Steve's suggestion and does work. – Otherness 24/2, 2014 at 6:42

This cannot be achieved with the standard HTTP request/response cycle. Basically what you are trying to do is make a "push" or "realtime" server. This can only be achieved with xhr-polling or websockets.

Code Example 1:

app.get('/path', function(req, res) {
   ...
   spw.stdout.on('data', function (data) {
      var str = data.toString();
      res.write(str + "\n");
   });
   ...
}

This code never sends an end signal and therefore will never respond. If you were to add a call to res.end() within that event handler, you will only get the first ping – which is the expected behavior because you are ending the response stream after the first chunk of data from stdout.

Code Sample 2:

spw.stdout.pipe(res);

Here stdout is flushing the packets to the browser, but the browser will not render the data chunks until all packets are received. Thus the reason why it waits 10 seconds and then renders the entirety of stdout. The major benefit to this method is not buffering the response in memory before sending — keeping your memory footprint lightweight.

Hearsay answered 21/2, 2014 at 14:31 Comment(3)

thanks for your answer.. actually I have a a res.end in spw.on('close'..., but just didn't write it down. I was thinking the same with you, but this answer confused me: #20357716 – Incipit 21/2, 2014 at 14:37

That answer is about streaming the data to the server in chunks to avoid buffering it all in the server first. Whether or not it is rendered in chunks is entirely up the the browser logic. – Hairline 21/2, 2014 at 15:47

Right — the server sends it all out on the pipe, but the browser will not close the connection until all packets are received. I changed my explanation to reflect this – don't know what I was thinking before! – Hearsay 21/2, 2014 at 15:58

Code Example 1:

Code Sample 2:

Recommended topics

Hot tags