IIS as a reverse proxy - compression of rewritten response from backend server

Question

I am implementing a reverse proxy for routing requests to a backend server.

Functionally everything works correctly, however I am concerned that all responses from the backend server are transferred to the client (web browser) without compression.

The setup is as follows:

• Backend server, not accessible for public, on an internal domain. Hosts a web application on https://internal.app
• Front web server with IIS 7.5, hosting the main public website and acting as a proxy for the backend server. The main site is at https://site.com.

I want to route all requests to https://site.com/app/WHATEVER to https://internal.app/WHATEVER in a way that is transparent to clients.

My current setup is based on URL Rewrite 2.0 and Application Request Routing IIS extensions. The general approach is based on guidelines from the following articles:

• Setting up a Reverse Proxy using IIS, URL Rewrite and ARR
• Reverse Proxy with URL Rewrite v2 and Application Request Routing

The relevant section of web.config of the site.com app:

<system.webServer>
    <rewrite>
        <rules>
            <rule name="Route the requests for backend app" stopProcessing="true">
                <match url="^app/(.*)" />
                <conditions>
                    <add input="{CACHE_URL}" pattern="^(https?)://" />
                </conditions>
                <action type="Rewrite" url="{C:1}://internal.app/{R:1}" />
                <serverVariables>
                    <set name="HTTP_ACCEPT_ENCODING" value="" />
                </serverVariables>
            </rule>
        </rules>
        <outboundRules>
            <rule name="RewriteBackendAbsoluteUrlsInResponse" preCondition="ResponseIsHtml1">
                <match filterByTags="A, Area, Base, Form, Frame, Head, IFrame, Img, Input, Link, Script" pattern="^http(s)?://internal.app(\:80)?/(.*)" />
                <action type="Rewrite" value="/app/{R:3}" />
            </rule>
            <rule name="RewriteBackendAbsoluteUrlsInRedirects" preCondition="ResponseIsHtml1">
                <match serverVariable="RESPONSE_LOCATION" pattern="^http(s)?://internal.app(\:80)?/(.*)" />
                <action type="Rewrite" value="/app/{R:3}" />
            </rule>
            <rule name="RewriteBackendRelativeUrlsInResponse" preCondition="ResponseIsHtml1">
                <match filterByTags="A, Area, Base, Form, Frame, Head, IFrame, Img, Input, Link, Script" pattern="^/(.*)" negate="false" />
                <conditions>
                    <add input="{URL}" pattern="^/app/.*" />
                </conditions>
                <action type="Rewrite" value="/app/{R:1}" />
            </rule>
            <rule name="RewriteBackendRelativeUrlsInRedirects" preCondition="ResponseIsHtml1">
                <match serverVariable="RESPONSE_LOCATION" pattern="^/(.*)" negate="false" />
                <conditions>
                    <add input="{URL}" pattern="^/app/.*" />
                </conditions>
                <action type="Rewrite" value="/app/{R:1}" />
            </rule>
            <preConditions>
                <preCondition name="ResponseIsHtml1">
                    <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                </preCondition>
            </preConditions>
        </outboundRules>
    </rewrite>
    <urlCompression dynamicCompressionBeforeCache="false" />
</system.webServer>

The problem is that as soon as I stop clearing the HTTP_ACCEPT_ENCODING server variable, each request which matches the above rule ends with the following error: HTTP Error 500.52 - URL Rewrite Module Error. Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded ("gzip").

I am aware of this thread and I have followed those instructions. I have set dynamicCompressionBeforeCache="false" as can be seen above, I have added the necessary registry entry and I have assured that the modules are in correct order in IIS.

However, this only seems to work only if the rewriting happens within one web app. If I remove the above rules and add a simple one (and respective outbound rules) to rewrite e.g. /x/WHATEVER to just /WHATEVER, all works perfectly without a need to clear HTTP_ACCEPT_ENCODING - the rule works and compression is enabled for the rewritten requests.

But as soon as I re-add my rule which rewrites the response to a different web app, and I don't clear the HTTP_ACCEPT_ENCODING header, the same error appears again.

From what I understand, if the rewriting involves another web app, there are more constraint on what can be done. E.g. URL rewriter must receive an uncompressed response from the backend server in order to be able to rewrite it using the outbound rules. I guess clearing HTTP_ACCEPT_ENCODING in this scenario is a must because of this.

However, I would expect that since the compression module is listed on top of the modules list, the final rewritten response should be compressed no matter where it originated from. It seems IIS makes some shortcuts and returns the response to the client bypassing the compression module. Or the HTTP_ACCEPT_ENCODING header is removed soon enough to completely disable compression (not only in server-to-server communication).

So finally, my question is: is there a way to compress those responses?

Palpitate

Answer 1

I have figured it out myself.

What needs to be done to get it working:

• Accept-Encoding header must be removed before routing the request to the backend server, so that the response can be rewritten using outbound rules
• the header must be restored by an additional accompanying outbound rule, so that it is present when the compression module kicks in before the response is sent to the client

I have decided to do it like this:

• add a new server variable to the rewrite rule to hold he original header sent by the client:

  <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />
  

  (I put it before the line which clears the HTTP_ACCEPT_ENCODING variable)

• add a new outbound rule:

  <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding">
    <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />
    <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />
  </rule>
  

  and an accompanying precondition:

  <preCondition name="NeedsRestoringAcceptEncoding">
    <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />
  </preCondition>
  

Works like a charm so far.

Palpitate

Glad you found this useful and thanks for linking back here! – Palpitate

Hm, I feel like I'm missing part of your formula. I've got all these things in place, and yet I'm still not seeing any compression for proxied content. What's the ordering of your IIS modules? Does it matter where HttpCacheModule and RewriteModule are in relation to DynamicCompressionModule and StaticCompressionModule? – Mcmichael

@Pauld'Aoust It's been a long time and I'm no longer involved in the project where I used this, so can't tell for sure, but the thread I linked to indicates that the order of modules does actually matter. – Palpitate

@JakubJanuszkiewicz thanks for taking the time to reply! I eventually went another route (looked at our app's code and found out that I didn't need response body rewriting, so I just left the HTTP_ACCEPT_ENCODING header as it is). – Mcmichael

While this solution addresses this 500.53 error described in the OP, it effectively is removing gzip compression from the response. That may be acceptable, but it should be noted that if gzip compression is the desired outcome, this solution does not provide that – Clerissa

I did all the above and kept getting a 500.53 error until i added HTTP_ACCEPT_ENCODING to the Allowed Server Variables in UrlRewrite config – Polydactyl

@Clerissa No, it doesn't remove compression from the final response. It removes compression from the request routed to the backend server, but then restores it (via the outbound rule) before the response is sent to the client. If it doesn't work for you, check the modules order - the outbound rule must be executed before the compression module executes, otherwise HTTP_ACCEPT_ENCODING is not set and you get no compression. – Palpitate

@JakubJanuszkiewicz If I have added into web.config than website not contain gZip. not sure what is wrong. If I remove both server variable than my site contain with gZip. Can you please let me what I miss? – Portmanteau

@KalpeshBoghara You'd need to provide more details about your website setup and the exact problem you're having. – Palpitate

@JakubJanuszkiewicz I have reorder modules and add one registry. So it's working fine please refer this: prntscr.com/oavri3 . Let me know if it's wrong – Portmanteau

Answer 2

To address the original poster's issue while still preserving gzip compressed responses, one simply needs to do the following:

1. update the registry on your public facing web server as such:

   a. For 64bit web sites, run the following in a command console with admin rights: reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\InetStp\Rewrite /v LogRewrittenUrlEnabled /t REG_DWORD /d 0

   b. For 32bit web sites, run the following in a command console with admin rights: reg add HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432node\Microsoft\Inetstp\Rewrite /v LogRewrittenUrlEnabled /t REG_DWORD /d 0

2. reset IIS

3. disable static compression

   a. to accomplish this, I inserted the following my web config: <urlCompression doStaticCompression="false" doDynamicCompression="true" dynamicCompressionBeforeCache="false" />

4. in the server node in IIS manager, double click on the Modules icon, then on the right, click "view ordered List" and verify that your static/dynamic compression modules are towards the top and the URL rewrite module is toward the bottom

Please Note

I see a lot of resolutions to this issue floating around the net where the HTTP_CONTENT_TYPE request header is being cleared out as part of the URL rewrite rule (including answers on this page). One should be aware that while this does resolve the original issue of the 500.52 error, gzip compression on the response is REMOVED. This may be the desired outcome, but if gzip compression required, the above solution will do the trick

Clerissa

Can you explain why this works and if it won't break in future versions of URL Rewrite? Aren't you using a side effect of the LogRewrittenUrlEnabled registry value to accomplish rewriting compressed responses? – Palpitate

Answer 3

PS: The below solution only works if you have control over your app server.

It's basically letting the web server do the compression, and let the app server do the heavy duty of what the app is supposed to do (without compression).

If you disable the compression on the app server, the response you get from app server is uncompressed. On the web server, you should enable the compression, so web server will honor the HTTP header "Accept-Encoding: gzip,deflate" while responding to client (browser).

This configuration will offload the CPU on the app server but will increase the network traffic between your web server and app server. If you are on the internal network, it doesn't have much performance impact.

Lorielorien

This worked for me, thanks. Here is the documentation page for changing these settings on IIS: iis.net/configreference/system.webserver/httpcompression – Dissension

Answer 4

Just adding a "serverVariables" tag in the rewrite rule did the trick for me:

<rewrite>
        <rules>
            <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                <match url="(.*)" />
                <action type="Rewrite" url="http://otherurl{R:1}" />

                <serverVariables>
                    <set name="HTTP_ACCEPT_ENCODING" value="" />
                </serverVariables>
            </rule>

       </rules>
</rewrite>

Ayres