NodeJS get a count of bytes of streaming file download

Question

In this code I stream a file from a url and save it to a file. Is there a way to also pipe it through something that will count the number of bytes piped? (Which would tell me the file size.)

  request.stream(url)
    .pipe(outputFile)

Is there some library that would do this by piping the download through it, or a simple way for me to do it myself?

Taiga

Answer 1

You can do it like this with request library:

const request = require('request');
const fs = require('fs');

var downloaded = 0;
request.get(url)
  .on('data', function(chunk){
    downloaded += chunk.length;
    console.log('downloaded', downloaded);
  })
  .pipe(fs.createWriteStream(fileName));

Also, you can check this link to learn how to do it without request package.

Update #1 (Pure NodeJS solution)

var http = require('http');
var fs = require('fs');

var download = function(url, dest, cb) {
  var file = fs.createWriteStream(dest);
  var downloaded = 0;
  var request = http.get(url, function(response) {
    response.pipe(file);
    response.on('data', function(chunk){
      downloaded += chunk.length;
      console.log(downloaded);
    })
    file.on('finish', function() {
      file.close(cb);
    });
  });
}

Altazimuth

Thanks for the request example. Would you mind providing an example also of how to do it without the request package based on that link you provided? – Taiga

file.on('finish'… should be file.on('end'… – Backset

In case of deflated pages, is there a way to count bytes before inflating? (any http module, or pure node) – Department

Answer 2

With typings and byteLength property:

const stream: ReadStream = file.createReadStream(); 
const chunks: Buffer[] = [];
let fileSize = 0;

stream.on('data', (chunk: Buffer) => {
  fileSize += chunk.byteLength;

  chunks.push(chunk);
});

stream.once('end', () => {
  const fileBuffer = Buffer.concat(chunks);

  console.log('equal values:', fileBuffer.byteLength === fileSize);
});

Granary