How do I stream response in express?

Question

I've been trying to get a express app to send the response as stream.

var Readable = require('stream').Readable;
var rs = Readable();


app.get('/report', function(req,res) {
    
    res.statusCode = 200;
    res.setHeader('Content-type', 'application/csv');
    res.setHeader('Access-Control-Allow-Origin', '*');

    // Header to force download
    res.setHeader('Content-disposition', 'attachment; filename=Report.csv');

    
    rs.pipe(res);

    rs.push("USERID,NAME,FBID,ACCOUNT,SUBSCRIPTION,PRICE,STATE,TIMEPERIOD\n");

    for (var i = 0; i < 10; i++) {
        rs.push("23,John Doe,1234,500,SUBSCRIPITON,100,ACTIVE,30\n");
    }

    rs.push(null);
});      

It does print in the console when I replace "rs.pipe(res)" by "rs.pipe(process.stdout)". But how to make it work in an express app?

Error: not implemented
    at Readable._read (_stream_readable.js:465:22)
    at Readable.read (_stream_readable.js:341:10)
    at Readable.on (_stream_readable.js:720:14)
    at Readable.pipe (_stream_readable.js:575:10)
    at line "rs.pipe(res);"

Mckibben

You need to subclass Readable, which needs to have a _read method. But why not just use res.write(...)? – Boiled

how would i use res.write() here? – Mckibben

Answer 1

You don't need a readable stream instance, just use res.write():

res.write("USERID,NAME,FBID,ACCOUNT,SUBSCRIPTION,PRICE,STATE,TIMEPERIOD\n");

for (var i = 0; i < 10; i++) {
    res.write("23,John Doe,1234,500,SUBSCRIPITON,100,ACTIVE,30\n");
}

res.end();

This works because in Express, res is based on Node's own http.serverResponse, so it inherits all its methods (like write).

Boiled

Will this still not be a problem if i add thousands of lines to the response? – Mckibben

No, because res is also a stream. – Boiled

Thank you. I've been trying to do it with the readable stream all day. – Mckibben

this will not stream,buffers till res.end() is called – Dentation

@chetandev your browser/client might buffer it, but Express or http doesn't. You can test this using a non-buffering TCP-client like nc/netcat. See this gist. – Boiled

I've found it is important, to initially send header first. Otherwise the client side won't receive streaming until res.end(). – Misanthrope

@BartoszRosa how can i do that – Osteogenesis

@Osteogenesis nodejs.org/api/http.html#responseflushheaders – Boiled

@Boiled any way to prevent browser/client from buffering it? I'd like to use axios but fetch is also fine. – Friendship

@ZulluBalti either implementation would need to support streaming responses (for fetch, look here; for axios, look here). – Boiled

Answer 2

I was able to get this to work.

• Put this code in your Express router.
• Open a web browser
• Go to http://yourdomain/yourrouterpath/stream

...

router.get('/stream', function (req, res, next) {
  //when using text/plain it did not stream
  //without charset=utf-8, it only worked in Chrome, not Firefox
  res.setHeader('Content-Type', 'text/html; charset=utf-8');
  res.setHeader('Transfer-Encoding', 'chunked');

  res.write("Thinking...");
  sendAndSleep(res, 1);
});


var sendAndSleep = function (response, counter) {
  if (counter > 10) {
    response.end();
  } else {
    response.write(" ;i=" + counter);
    counter++;
    setTimeout(function () {
      sendAndSleep(response, counter);
    }, 1000)
  };
};

Expand

Any ideas why this doesnt work with Content-Type: text/plain; charset=utf-8 ?? – Harlequinade

Wow thank you so much for this answer. I was setting the content type to text/plain and couldn't figure out for the life of me why it wasn't streaming. You can also just not set the header, start writing, & it will work. – Linnet

MDN notes the use of 'Transfer-Encoding' is disallowed in HTTP/2 (except in one case, which is not this one): developer.mozilla.org/en-US/docs/Web/HTTP/Headers/… – Shelby

it's probably still delivered streaming to the client, but without the charset specified, the browser probably waits to receive a bunch of data to guess what the charset is before rendering any of it – Autograft

Answer 3

I needed to stream a response in express in order to work with tar-stream. Here is how I did it in case it helps anyone.

The requests are for a single file from a tar file stored on the server.

const fs = require("fs"),
   tar = require("tar-stream");

app.get("/fileFromTar/*", (req, res) => {
   const fileWanted = req.params[0],
      readStream = fs.createReadStream('myTarFile.tar'),
      extractor = tar.extract();

   extractor.on('entry', (header, stream, next) => {
      stream.on('end', next);

      if (header.name === fileWanted) {
         const { size } = header;
         res.set({
           "Content-Type": 'audio/flac', // or whichever one applies
           "Content-Length": size,
           "Content-Range": `bytes 0-${size}/${size}`
         });
         stream.pipe(res);
      }
      else stream.resume();
   });
   readStream.pipe(extractor);
});

Garlinda

Answer 4

While most of the answers are correct on this thread in the context of Streaming data, but if you want to make the streamed data work with either Postman or EventSource (Browser interface), you would need to write the data on response with data field and also set few headers as demonstrated below:

app.get(`/stream`, (req: Request, res: Response) => {
    
    // setting below headers for Streaming the data
    res.writeHead(200, {
        'Content-Type': "text/event-stream",
        'Cache-Control': "no-cache",
        'Connection': "keep-alive"
    });
    
    let value = 0;
    setInterval(() => {
        res.write(`data: ${JSON.stringify({count: value++})}\n\n`); // "data:" is important here
    }, 1000);
});

Transcribe