How to enable gzip HTTP compression on Windows Azure dynamic content

Question

I've been trying unsuccessfully to enable gzip HTTP compression on my Windows Azure hosted WCF Restful service which returns JSON only from GET and POST requests.

I have tried so many things that I would have a hard time listing all of them, and I now realise I have been working with conflicting information (regarding old version of azure etc) so think it best to start with a clean slate!

I am working with Visual Studio 2008, using the February 2010 tools for Visual Studio.

So, according to the following link..

.. HTTP compression has now been enabled. I've used the advice at the following page (the URL compression advice only)..

http://blog.smarx.com/posts/iis-compression-in-windows-azure

<urlCompression doStaticCompression="true" 
         doDynamicCompression="true"
         dynamicCompressionBeforeCache="true" 
/>

.. but I get no compression. It doesn't help that I don't know what the difference is between urlCompression and httpCompression. I've tried to find out but to no avail!

Could, the fact that the tools for Visual Studio were released before the version of Azure which supports compression, be a problem? I have read somewhere that, with the latest tools, you can choose which version of Azure OS you want to use when you publish ... but I don't know if that's true, and if it is, I can't find where to choose. Could I be using a pre-http enabled version?

I've also tried blowery http compression module, but no results.

Does any one have any up-to-date advice on how to achieve this? i.e. advice that relates to the current version of the Azure OS.

Cheers!

Steven

Update: I edited the above code to fix a type in the web.config snippet.

Update 2: Testing the responses using the whatsmyip URL shown in the answer below is showing that my JSON responses from my service.svc are being returned without any compression, but static HTML pages ARE being returned with gzip compression. Any advice on how to get the JSON responses to compress will be gratefully received!

Update 3: Tried a JSON response larger than 256KB to see if the problem was due to the JSON response being smaller than this as mentioned in comments below. Unfortunately the response is still un-compressed.

Reserved

Shared hosts should have gzip enabled by default for all json web services. Sure it will put a little more cpu load on the server, but the benefit of smaller packets traveling across the net far outweighs the cost - IMO. Not all of us have fiber running to our homes. – Eastman

Answer 1

Well it took a very long time ... but I have finally solved this, and I want to post the answer for anyone else who is struggling. The solution is very simple and I've verified that it does definitely work!!

Edit your ServiceDefinition.csdef file to contain this in the WebRole tag:

    <Startup>
      <Task commandLine="EnableCompression.cmd" executionContext="elevated" taskType="simple"></Task>
    </Startup>

In your web-role, create a text file and save it as "EnableCompression.cmd"

EnableCompression.cmd should contain this:

%windir%\system32\inetsrv\appcmd set config /section:urlCompression /doDynamicCompression:True /commit:apphost
%windir%\system32\inetsrv\appcmd set config  -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/json; charset=utf-8',enabled='True']" /commit:apphost

.. and that's it! Done! This enables dynamic compression for the json returned by the web-role, which I think I read somewhere has a rather odd mime type, so make sure you copy the code exactly.

Reserved

Is there a way to only execute these statements when running on real azure? – Corry

Nevermind found the answer here (requires SDK 1.5) blog.smarx.com/posts/… – Corry

Is this still relevant in the current SDK? I think I have it configured correctly, but I get an "unable to start debugging" error when running locally, no matter what the contents of the cmd file are. – Gotthelf

I've implemented #11811224 as Brian suggests in his update so that the instances can be recycled. – Destroyer

You have the Startup Task taskType set to taskType="simple", is it necessary the script runs in simple mode or can it use background as well? (Clarification on the difference can be found here) – Crossgarnet

@Steven, Do you know how can we compress Images from azure? – Haulm

Is this still the preferred way to do this? I can't help but feel that running a batch file at deployment time is a hack. I found this article azure.microsoft.com/en-us/blog/… but am not entirely sure on how to implement it at the moment. – Weatherly

As near as I can tell, they haven't added anything more convenient for Cloud Services (which are almost deprecated in Azure these days as a hosting option, and have not seen any improvements in general for years). Also note that @TheSenator 's link talks about Azure Web Sites, not Azure Cloud Services. – Tralee

Answer 2

Well at least I'm not alone on this one - and it's still a stupid PITA almost a year later.

The problem is a MIME type mismatch. WCF returns JSON response with Content-Type: application/json; charset=UTF-8. The default IIS configuration, about halfway down that page, does not include that as a compressible MIME type.

Now, it might be tempting to add an <httpCompression> section to your web.config, and add application/json to that. But that's just a bad way to waste a good hour or two - you can only change the <httpCompression> element at the applicationHost.config level.

So there are two possible solutions. First, you could change your WCF response to use a MIME type that is compressible in the default configuration. text/json will work so adding this to your service method(s) will give you dynamic compression: WebOperationContext.Current.OutgoingResponse.ContentType = "text/json";

Alternatively, you could change the applicationHost.config file using appcmd and a startup task. This is discussed (among other things) on this thread. Note that if you add that startup task and run it in the dev fabric, it will work once. The second time it will fail because you already added the configuration element. I ended up creating a second cloud project with a separate csdef file, so that my devfabric would not run that startup script. There are probably other solutions though.

Update

My suggestion for separate projects in the previous paragraph is not really a good idea. Non-idempotent startup tasks are a very bad idea, because some day the Azure fabric will decide to restart your roles for you, the startup task will fail, and it'll go into a recycle loop. Most likely in the middle of the night. Instead, make your startup tasks idempotent as discussed on this SO thread.

Monika

Important to note that <httpCompression> must be added at the applicationHost.config level. I bet that many people - myself included - have wasted time trying to get it working by modifying web.config. – Portecochere

Answer 3

To deal with local development fabric having issues after first deploy, I added the appropriate commands to the CMD file to reset config. In addition, I'm setting compression level here specifically, since it appears to default to zero in some (all?) cases.

REM Remove old settings - keeps local deploys working (since you get errors otherwise)
%windir%\system32\inetsrv\appcmd reset config -section:urlCompression
%windir%\system32\inetsrv\appcmd reset config -section:system.webServer/httpCompression 

REM urlCompression - is this needed?
%windir%\system32\inetsrv\appcmd set config -section:urlCompression /doDynamicCompression:True /commit:apphost
REM Enable json mime type
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/json; charset=utf-8',enabled='True']" /commit:apphost

REM IIS Defaults
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='text/*',enabled='True']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='message/*',enabled='True']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/x-javascript',enabled='True']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='*/*',enabled='False']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"staticTypes.[mimeType='text/*',enabled='True']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"staticTypes.[mimeType='message/*',enabled='True']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"staticTypes.[mimeType='application/javascript',enabled='True']" /commit:apphost
%windir%\system32\inetsrv\appcmd set config -section:system.webServer/httpCompression /+"staticTypes.[mimeType='*/*',enabled='False']" /commit:apphost

REM Set dynamic compression level to appropriate level.  Note gzip will already be present because of reset above, but compression level will be zero after reset.
%windir%\system32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"[name='deflate',doStaticCompression='True',doDynamicCompression='True',dynamicCompressionLevel='7',dll='%%Windir%%\system32\inetsrv\gzip.dll']" /commit:apphost
%windir%\system32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression -[name='gzip'].dynamicCompressionLevel:7 /commit:apphost

Xerxes

Answer 4

This article from MS is their how to script for JSON http://msdn.microsoft.com/en-us/library/windowsazure/hh974418.aspx.

It deals with many of the issues mentioned e.g. being able to handle Azure recycle etc

Defective

Answer 5

Just had an issue with this regarding the error type 183, and I found a solution. So if anybody else is experiencing this here goes:

Here's the error I got:

User program "F:\approot\bin\EnableCompression.cmd" exited with non-zero exit code 183. Working Directory is F:\approot\bin.

And here's the code that fixed it for me:

REM   *** Add a compression section to the Web.config file. ***
%windir%\system32\inetsrv\appcmd set config /section:urlCompression /doDynamicCompression:True /commit:apphost >> "%TEMP%\StartupLog.txt" 2>&1

REM   ERRORLEVEL 183 occurs when trying to add a section that already exists. This error is expected if this
REM   batch file were executed twice. This can occur and must be accounted for in a Windows Azure startup
REM   task. To handle this situation, set the ERRORLEVEL to zero by using the Verify command. The Verify
REM   command will safely set the ERRORLEVEL to zero.
IF %ERRORLEVEL% EQU 183 DO VERIFY > NUL

REM   If the ERRORLEVEL is not zero at this point, some other error occurred.
IF %ERRORLEVEL% NEQ 0 (
   ECHO Error adding a compression section to the Web.config file. >> "%TEMP%\StartupLog.txt" 2>&1
   GOTO ErrorExit
)

REM   *** Add compression for json. ***
%windir%\system32\inetsrv\appcmd set config  -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/json; charset=utf-8',enabled='True']" /commit:apphost >> "%TEMP%\StartupLog.txt" 2>&1
IF %ERRORLEVEL% EQU 183 VERIFY > NUL
IF %ERRORLEVEL% NEQ 0 (
   ECHO Error adding the JSON compression type to the Web.config file. >> "%TEMP%\StartupLog.txt" 2>&1
   GOTO ErrorExit
)

REM   *** Exit batch file. ***
EXIT /b 0

REM   *** Log error and exit ***
:ErrorExit
REM   Report the date, time, and ERRORLEVEL of the error.
DATE /T >> "%TEMP%\StartupLog.txt" 2>&1
TIME /T >> "%TEMP%\StartupLog.txt" 2>&1
ECHO An error occurred during startup. ERRORLEVEL = %ERRORLEVEL% >> "%TEMP%\StartupLog.txt" 2>&1
EXIT %ERRORLEVEL%

Solution found at http://msdn.microsoft.com/en-us/library/azure/hh974418.aspx

Grapheme

Answer 6

Yes, you can choose the OS you want, but by default, you'll get the latest.

Compression is tricky. There are lots of things that can go wrong. Are you by chance doing this testing behind a proxy server? I believe IIS by default doesn't send compressed content to proxies. I found a handy tool to test whether compression is working when I was playing with this: http://www.whatsmyip.org/http_compression/.

It looks like you have doDynamicCompression="false"... is that just a typo? You want that to be on if you're going to get compression on JSON you return from a web service.

Margitmargo

Hey! Thanks for your quick reply! That was a typo, the line should read: <urlCompression doStaticCompression="true" doDynamicCompression="true" dynamicCompressionBeforeCache="true" /> I used the whatsmyip URL you suggested to do some testing. My JSON responses (from my service.svc) are being returned without any compression, but static HTML pages ARE being compressed. Do you have any suggestions of where I can go from here? Thanks again – Reserved

Try generating really big JSON. According to msdn.microsoft.com/en-us/library/ms690689(VS.90).aspx, the default minSize setting is 256 KB. – Margitmargo

I created a fake response that was 280KB in size, and tested it at whatsmyip ... unfortunately it's still un-compressed. – Reserved

I'm pretty sure you can enable GZIP compression by making a custom IHTTPModule ( west-wind.com/weblog/posts/2007/Jan/23/… ) and adding it to Azure ( blogs.msdn.com/b/tom/archive/2011/02/18/… ) – Zaxis