What is back pressure on stream processing?

Question

I started learning NodeJS, and streams seems to be something that people use a lot. In most of the documentation that I had read there are mentions of the "back-pressure problem" that occurs when you are processing big sized files but I haven't found a clear explanation of what exactly this problem is. Also I have read that using pipes can help with this problem, but how exactly do pipes fix the back-pressure problem?

Thanks for any explanation in advance.

Endocrinology

Imagine you had something like readStream.pipe(writeStream). Back pressure is when the writeStream is unable to consume the data as quickly as the readStream is pushing data to it. This article does a decent job of explaining the issue and how you might handle it. – Ordonez

Thanks a lot, that article was really helpful. – Endocrinology

@idbehold, link doesn't work.Do you have other link? – Gefell

@Gefell github.com/creationix/howtonode.org/blob/master/articles/… – Ordonez

Thanks for the link. – Gefell

The link is also archived: archive.is/KZabP – Ellette

Answer 1

Backpressure is when you write to a stream faster than the other process can handle/consume Using Pipes, you can control the flow, pause and resume the stream; Here is an example implementing backpressure in nodejs

var http = require('http'),
    fs = require('fs');

var server = http.createServer(function(request, response) {
  var file = fs.createWriteStream('upload.jpg'),
      fileBytes = request.headers['content-length'],
      uploadedBytes = 0;

  request.on('data', function(chunk) {
    uploadedBytes += chunk.length;
    var progress = (uploadedBytes / fileBytes) * 100;
    response.write('progress: ' + parseInt(progress, 10) + '%\n');

    var bufferOK = file.write(chunk);

    if (!bufferOK) {
      request.pause();
    }
  });

  file.on('drain', function() {
    request.resume();
  });

  request.on('end', function() {
    response.end('upload complete\n');
  });

});

server.listen(8080);

Ben foster solution - Souce: https://gist.github.com/benfoster/9543337

Emmetropia

how does client must react on pausing channel? can it continue throwing away requests assuming underlying transport system controls everything (actually not cause somewhere is limited buffer/queue)? – Iamb