node express/request: piping a POST request with body parsing

Question

I'm attempting to pipe a request for handling by a remote server, along the following lines:

var destination = request(url);
req.pipe(destination).pipe(res);

This works just fine for GET requests. But for POST requests I'm struggling. An important point to note, I think, is that for POST requests I'm using a body parser before my POST route handler in order to extract the body from the POST request... it's just a basic text body parser because the body contains plain text:

var postRouteHandler = someFunction;
var bodyParser = require('body-parser');
var textParser = bodyParser.text({
    limit: '50kb'
});
app.use('/postRoute', [textParser, postRouteHandler]);

From this issue and this issue it seems to me that doing any processing on the POST request before you pipe it will cause a problem. Indeed, when I remove the parser, the piping seems to work OK.

The problem is that I need to examine the body first, to do some initial processing and then to determine whether or not to pipe the request on to the remote server at all. So I need to parse the body before piping.

Is there any way around this problem?

Somatic

Answer 1

The issue is that with streams (like req), once you've read it you can't reset it.

Because body-parser has read the stream already, piping it won't work because that will try to read the stream again (which at that point has been exhausted).

A workaround would be take the text read by body-parser, and create a minimal req "clone" in order for request to be able to pass the request along:

var intoStream = require('into-stream');
var bodyParser = require('body-parser');
var textParser = bodyParser.text({ limit: '50kb' });

var postRouteHandler = function(req, res) {
  let text = req.body;
  if (! shouldPipe(text)) {
    return res.sendStatus(400); // or whatever
  }

  // Here's where the magic happens: create a new stream from `text`,
  // and copy the properties from `req` that `request` needs to pass
  // along the request to the destination.
  let stream     = intoStream(text);
  stream.method  = req.method;
  stream.headers = req.headers;

  // Pipe the newly created stream to request.
  stream.pipe(request(url)).pipe(res);
};
app.use('/postRoute', [textParser, postRouteHandler]);

Cloudy

Learn from the master! Genius! Not only a great solution, but also a clear and succinct explanation of why the problem happens. You are indeed the master @robertklep! – Somatic

... just one thing... should it be request.post(url) rather than just request(url)? – Somatic

I'm pretty sure that request will use req.method/stream.method to determine what type of request to perform, so you don't have to be explicit about it. – Cloudy

I cant seem to be able to use into-stream (node with version 6.14, can I use another 3rd party instead? is there another option?) – Uria

@OmriShneor into stream is a package on npm npmjs.com/package/into-stream – Luxemburg

Answer 2

You should create your own middleware if the parser is causing the problem. In all honesty you may want to parse the body a different way than the body parser.

Given the limited knowledge about what the rest of your project is doing you could create middleware to hook the body parser middleware and just send a clone of request. This however is not very efficient but will work and may help point you in the right direction.

var postRouteHandler = someFunction;
var bodyParser = require('body-parser');
var textParseMiddleware = function (req, res, next) {

  var requestclone = _.clone(req);
  var textParser = bodyParser.text({
    limit: '50kb'
  });

  textParser(requestclone, res, function(){
    console.log('examine the body here', requestclone.body);
  });


}

app.use('/postRoute', [textParserMiddleWare, postRouteHandler]);

I have not tested the above code.

Homespun

Answer 3

Need to send a huge large data via POST, I was getting Error: socket hang up / code: 'ECONNRESET'. Was able to resolve it via increasing limit in body parser.

var bodyParser = require('body-parser'); app.use(bodyParser.urlencoded({extended: true})); app.use(bodyParser.json({limit:'50mb'}));

Resnatron

How does this answer the question? Kindly ensure you answer relates to the question asked. – Luxemburg