Websockets reverse proxy in IIS 8

Question

I'm attempting to connect to a websockets server (websockify) through a reverse proxy on IIS. The IIS and websockets server reside on the same physical server (Windows Server 2012 R2, IIS 8.5, ARR 3, Websockets enabled). I've seen a few questions about this and it's suggested this should work with IIS 8 and ARR 3, but no actual solutions as yet. I have some experience with http/https reverse proxies in IIS, but this is my first attempt working with websockets.

For example:

The original url: ws://10.2.1.10/websockify

The reverse proxy needs to translate this to: ws://10.2.1.10:5901/websockify

Overly general sample rule in web.config:

<rewrite>
     <rules>               
        <rule name="WS reverse proxy" stopProcessing="true"> 
          <match url="(.*)" />
          <conditions> <add input="{CACHE_URL}" pattern="^(.+)://" /> 
          </conditions> 
          <action type="Rewrite" url="{C:1}://10.2.1.10:5901/websockify"/>       
        </rule>
     </rules>
</rewrite>

Per the Failed Request Trace, the url appears to be translated, but for some reason it doesn't reach the websocket server at 10.2.1.10:5901.

The end goal is to incorporate noVNC/websockify to provide browser based client access to multiple VNC servers on the network. Any help understanding how to reverse proxy the websockets is appreciated.

Audriaaudrie

Answer 1

I had been trying to accomplish this same thing on IIS 8.5 with ARR 3.0, and eventually found the problem. According to Microsoft's Erez Benari, this is possible:

WebSocket support requires the WebSocket feature to be installed on IIS, but does not require any other configuration or action. Install the feature using the Server Manager Add Roles and Features, and once that is complete, ARR 3.0 will handle the requests appropriately.

As a test, I set up a Node.js server for WebSocket:

const WebSocketServer = require('ws');
const wss = new WebSocketServer({ port: 3011 });
function sendWSMessage(msg) {
    wss.clients.forEach((client) => {
        client.send(msg);
    });
}
setInterval(function() {
    sendWSMessage('hello client');
}, 3000);

Along with a simple test page:

var websock = new WebSocket('ws://localhost:3011');
websock.onmessage = function (event) {
    console.log(event.data);
};
websock.onopen = function (event) {
  websock.send("hello server"); 
};

Then, I set up an ARR reverse proxy on my local machine, with the following in a web.config file of a "wstest" directory on localhost:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
    <rewrite>
        <rules>
            <rule name="WebSocketTestRule" stopProcessing="true">
                <match url=".*" />
                <conditions>
                    <add input="{CACHE_URL}" pattern="^(.+)://" />
                </conditions>
                <action type="Rewrite" url="{C:1}://localhost:3011/" />
            </rule>
        </rules>
    </rewrite>
</system.webServer>
</configuration>

This should forward all traffic for //localhost/wstest to a Node.js server on port 3011. The Node server works when I directly connect to it via ws://localhost:3011. When I try to connect through the proxy via ws://localhost/wstest, the request makes it through to the Node.js server, the upgrade occurs, and the connection is made.

Chrome sends:

GET ws://localhost/wstest HTTP/1.1
Host: localhost
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Origin: file://
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Sec-WebSocket-Key: 4ufu8nAOj7cKndASs4EX9w==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

The Node.js server receives:

cache-control: no-cache
connection: upgrade
pragma: no-cache
upgrade: Websocket
accept-encoding: gzip, deflate, sdch
accept-language: en-US,en;q=0.8
host: localhost:3011
max-forwards: 10
user-agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
origin: file://
sec-websocket-version: 13
sec-websocket-key: fBkTwAS9d/unXYKDE3+Jjg==
sec-websocket-extensions: permessage-deflate; client_max_window_bits
x-original-url: /wstest
x-forwarded-for: [::1]:54499
x-arr-log-id: a0b27458-9231-491d-b74b-07ae5a01c300

The Node.js server responds with:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-Websocket-Accept: yep8mgQACAc93oGIk8Azde4WSXk=
Sec-WebSocket-Extensions: permessage-deflate

And finally Chrome receives:

HTTP/1.1 101 Switching Protocols
Upgrade: Websocket
Sec-WebSocket-Accept: CBSM8dzuDoDG0OrJC28nIqaw/sI=
Sec-WebSocket-Extensions: permessage-deflate
X-Powered-By: ARR/3.0
Connection: Upgrade
X-Powered-By: ASP.NET
Date: Fri, 10 Jun 2016 21:16:16 GMT
EndTime: 17:16:16.148
ReceivedBytes: 0
SentBytes: 0

So now they are connected. This all looks good, the only noticeable difference being that the Sec-WebSocket-Key and Sec-WebSocket-Accept is changed in both directions by either IIS or the ARR proxy.

But... no WebSocket frames ever make it through the proxy! When Chrome receives positive feedback on its upgrade request, it sends its WebSocket message frame, and it is then sitting and waiting for messages from the server. The Node.js server sends its frames, and no error occurs, but they are never received by Chrome. The message that Chrome sent is never received by Node.js. It appears that ARR/IIS is dropping the WebSocket frames in both directions.

Notice how Chrome is telling the server that it supports the permessage-deflate extension, which is a WebSocket extension for per-message compression. The server is responding that it also supports permessage-deflate, so when they browser and server send their messages to each other, they use this compression extension. HOWEVER, the guy in the middle, ARR, apparently does NOT support this compression! By turning off support for permessage-deflate on the server, the actual WebSocket frames can now pass through the proxy flawlessly:

const wss = new WebSocketServer({ port: 3011, perMessageDeflate: false });

I think the issue is that ARR 3.0 does not support the Sec-Websocket-Extensions header, so it is allowing the header to simply pass through. But allowing this header to be negotiated between the client and the server is wrong, because ARR is not involved in the negotiation and has no way of telling the two parties that it does not support passing compressed messages. Hopefully someday, ARR will be able to properly handle extensions by negotiating between itself and the client, and then doing a separate negotiation between itself and the server. As it stands now, it simply has the client and server negotiating with each other, which results in this error.

Cist

I'm using IIS 10 on Server 2016 with ARR 3.0 and (once having learned that I need to install the Websockets feature) I'm getting two way websocket communication across the proxy to chrome, using https (and which includes the permessage-deflate extension in Sec-Websocket-Extentions). I did not try your test exactly (since I'm testing with my own application) but it seems to work. Perhaps fixed in IIS 10? – Vowelize

@Vowelize Thanks for your experience. I agree: If the traffic is being compressed between the browser and the proxy, then ARR 3.0 on IIS 10 must support the permessage-deflate extension. – Cist

@Vowelize - Your comment just made my life so much better. I had ARR 2.5 installed and was having all sorts of problems with the upgrade request... I installed ARR 3.0 and everything worked as expected. – Numerate

I'm using pretty much this configuration but something you don't mention is how well your socket are closed. For mine, adding IIS causes ECONNRESET in the server app. Somehow the close handshake isn't passed through? #64790497 – Cannikin

Answer 2

As stated by @jowo, IIS8/8.5 don't seem to support Sec-Websocket-Extensions. That being said, the workaround I applied is to simply rewrite the server variable in question. First add HTTP_SEC_WEBSOCKET_EXTENSIONS to the allowed server variable list, then add it to your rule:

<serverVariables><set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="" /></serverVariables>

That way, the destination won't receive the troublesome permessage-deflate :)

Abbacy

Answer 3

In Server Manager, make sure you enabled Web Socket Protocol which is available under Web Server (IIS) > Web Server > Application Development

Strepitous

Answer 4

Just an update a couple years later after having spent too much time on this issue, trying to reverse proxy a couple gradio apps with subdomains and custom SSL certificates. As far as I can tell, the perMessageDeflate is still unsupported on IIS 10 + ARR 3.0 and the HTTP_SEC_WEBSOCKET_EXTENSIONS header override still applies.

However, setting the value to an empty string will get the header to fail parsing with a regex exception within the gradio websocket stack. The following value did the trick:

<serverVariables>
   <set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="nodata" />
</serverVariables>

Protochordate

Answer 5

The TLDR is to:

1. url redirect the ws and wss protocols correctly
2. disable the compression in the node server socket io, passing the option perMessageDeflate. disable compression as APR 3 In IIS Proxy does not support it

const io = require("socket.io")(server, {
  transports: ["websocket", "polling"],
  perMessageDeflate: false 
});

Honna

Care to elaborate on what "correctly" means in #1 there? – Cannikin