Forward every request with node-http-proxy

Question

I'm trying to setup a node-http-proxy that just forwards requests. In the end this proxy should inject javascript in every website I visit through the browser..

Right now, most pages are forwarded and displayed correctly, but some, like posterkoenig.ch or verkehrsclub.ch are returning either a blank page or there is an error on the page. Both sites work well without the proxy in place. What do I have to change, or what am I missing that gets not forwarded correctly?

Im very new to nodejs and not even completely sure if my approach should work or not.

Here is what I've got so far:

var httpProxy = require('http-proxy');
var url = require('url');

httpProxy.createServer(function(req, res, proxy) {

  var urlObj = url.parse(req.url);

  proxy.proxyRequest(req, res, {
    host: urlObj.host,
    port: 80,
    changeOrigin: true,
    enable : { xforward: true }
  });
}).listen(9000, function () {
  console.log("Waiting for requests...");
});

---

Update

As suggested by @robertklep I removed changeOrigin and redefined req.headers.host and also req.headers.url

posterkoenig.ch:

Now throws:

An error has occurred: 
{"code":"ENOTFOUND","errno":"ENOTFOUND","syscall":"getaddrinfo"}

verkehrsclub.ch:

The frontpage works now but subpages still throw a error on the page.

var httpProxy = require('http-proxy');
var url = require('url');

httpProxy.createServer(function(req, res, proxy) {

  var urlObj = url.parse(req.url);

  req.headers['host'] = urlObj.host;
  req.headers['url'] = urlObj.href;

  proxy.proxyRequest(req, res, {
    host: urlObj.host,
    port: 80,
    enable : { xforward: true }
  });
}).listen(9000, function () {
  console.log("Waiting for requests...");
});

Butyrate

were you able to do this for HTTPS sites? – Verdugo

Basically it should be possible.. I haven't tried it, but the documentation of node-http-proxy says it supports https. But your node app would have to run an https server as well, so you need a valid https certificate and key to make it work. Then you should be able to use a slight variation of the script above with an https server. Hope that helps! – Butyrate

How have you configured your browser to use this proxy? Just entered localhost:9000 as the web proxy server address? – Missioner

Answer 1

Your first problem is related to changeOrigin: that will send a Host header to the remote server which includes a port number, and both sites you mention can't handle that.

Instead, try this:

req.headers.host = urlObj.host;
req.url          = urlObj.path;
proxy.proxyRequest(req, res, {
  host: urlObj.host,
  port: 80,
  enable : { xforward: true }
});

As for your other problem, I think it might be related to websites that don't serve their content as UTF-8 (which is the encoding that .toString() will use if you don't pass it an encoding). Does it happen always, or just with some sites?

FWIW, harmon is a middleware for node-http-proxy which provides a nice way of rewriting responses. It might be an overkill for your situation, but it might also solve your problem.

EDIT: here's a minimal example that seems to work just fine for both posterkoenig.ch and www.verkehrsclub.ch (homepages as well as subpages):

var httpProxy = require('http-proxy');
var url       = require('url');

httpProxy.createServer(function(req, res, proxy) {
  var urlObj = url.parse(req.url);

  req.headers.host  = urlObj.host;
  req.url           = urlObj.path;

  proxy.proxyRequest(req, res, {
    host    : urlObj.host,
    port    : 80,
    enable  : { xforward: true }
  });
}).listen(9000, function () {
  console.log("Waiting for requests...");
});

Lawmaker

Thanks for your time! I tried this, but when I remove the <code>changeOrigin</code> flag, posterkoenig.ch returns An error has occurred: {"code":"ENOTFOUND","errno":"ENOTFOUND","syscall":"getaddrinfo"}. I then also redefined the headers.url.. with this, the verkehrsclub.ch frontpage appears correctly but every subpage is still broken.. I updated my code in the question – Butyrate

The error you're getting for postkoenig.ch suggests a DNS error, which is strange (what's the value of urlObj.host in that case?). Also, I edited my answer because there's another issue I spotted which might cause your second problem. – Lawmaker

Thanks, the req.url = urlObj.path; did the job! – Butyrate

I tried the minimal example, but it only pays attention to requests coming from 127.0.0.1. How do I make it listen to requests coming from an external source as well? – Dira

@SzerémiAttila I don't think that's something to do with http-proxy, I tested it (run the proxy on my Mac, access it via my Linux server) and it works as-is. Perhaps you have a firewall that's blocking incoming connections? – Lawmaker

@Lawmaker it's probably a firewall as I tested on a different server that I had more control over and it worked there. Thanks! – Dira

What url do you hit to make the node proxy code do it's magic? – Starch

This doesn't work anymore, as now it is mandatory to pass some kind of a target or a forward field in the options. Running the above code gives me Must provide valid url for Target. @Butyrate – Missioner

@AyushGoel instead of setting the host and port options, try using target : req.url – Lawmaker

No that doesn't work. See this . Options is a dictionary where target/forward is one of the required fields. – Missioner

I haven't been able to make it work for the life of me. @robertklep. If you have any idea how this can work with the latest library code, can you post the answer here. Thanks a lot – Missioner