How to host static content pre-compressed in apache?

Question

I host a JavaScript game which basically consists of an .html and a .data file. If I compress them with gzip, their size shrinks to 25%. So I want to do that.

I'm not 100% sure, but I think using mod_gzip or mod_deflate does the compression on-the-fly, wasting cpu time all the time because the Content doesn't Change.

So I'ld like to precompile the Content. Therefore, I put a .gz next to the uncompressed files and put rewrite rules in .htaccess:

RewriteEngine on 
# If client accepts compressed files 
RewriteCond %{HTTP:Accept-Encoding} gzip 
# and if compressed file exists 
RewriteCond %{REQUEST_FILENAME}.gz -f 
# send .html.gz instead of .html 
RewriteRule ^(.+)\.(html|css|js|data)$ $1.$2.gz [T=text/$2,E=GZIP:gzip,L] 
Header set Content-Encoding gzip env=GZIP 

The Redirect is working, I can request game.html and actually get deliviered game.html.gz. However, the browser doesn't just display it. Instead, it asks me where to save the file. How can I fix that? Or maybe there is another way to achieve my Goal?

Undermanned

Seems like apache does not respect the T modifier. What content-type does the server actually return? – Moonstruck

It's hard for me to find out. Firebug doesn't Report the Content-type when ff wants to save the file - or I don't know how to see it. – Undermanned

Answer 1

This is how i fixed once the same problem.

Add new types in .htaccess:

AddEncoding gzip .jsgz .cssgz .htmlgz .datagz
AddType application/javascript .jsgz
AddType text/css .cssgz
AddType text/html .htmlgz       
AddType text/plain .datagz

This was done this way because AddType instruction didn't accept extensions in the form .html.gz.

Then modify your rewrite rule:

RewriteRule ^(.+)\.(html|css|js|data)$ $1.$2gz [L] 

And finally rename your files. Remove dots from .html.gz, .js.gz and so on.

The full .htaccess would look like this:

AddEncoding gzip .jsgz .cssgz .htmlgz .datagz
AddType application/x-javascript .jsgz
AddType text/css .cssgz
AddType text/html .htmlgz       
AddType text/plain .datagz

RewriteEngine on 
# If client accepts compressed files 
RewriteCond %{HTTP:Accept-Encoding} gzip 
# and if compressed file exists 
RewriteCond %{REQUEST_FILENAME}gz -f 
# send .html.gz instead of .html 
RewriteRule ^(.+)\.(html|css|js|data)$ $1.$2gz [L] 

Moonstruck

Is it crucial to have the extensions named like that? Because I tried your solution with replacing .htmlgz with .html.gz and so on and again, it asks to save the file instead of displaying it. – Undermanned

It actually seems to be crucial for some reason. If I rename my .html.gz files to .htmlgz and so forth, using your suggested .htaccess works. – Undermanned

this solution really works on a live project for several years. If AddType would support complex extensions like .html.gz you could not rename the files. – Moonstruck

Works for me too. I have CSS and JS minified/gzipped and it's great. Thanks – Nervine

The only issue remaining with this solution is that it doesn't set Vary:accept-encoding , which potentially breaks intermediate caches. – Sweetsop

@MattyK When the RewriteCond that uses %{HTTP:Accept-Encoding} matches, Apache should (according to the docs) automatically add that Vary header. – Zuleika

I have been looking for this kind of solution for some time now. Thank you! – Jeopardous

this works #9077252 – Ward

Answer 2

The first question you should ask yourself is, is there any point in doing this? Do you notice too high cpu load and/or performance difference because of this? My guess would be that you are probably not running into this problem :)

Regardless though, there are multiple ways of fixing your problem.

1. Probably the best option for you, use a CDN. They are designed for fast delivery of static files and will make it fast for people in a different geographical area as well as people close to your server. Also, in my experience CDN's are usually much cheaper than your own bandwidth will be.

2. Use Nginx. For hosting static files much faster and has support for pre generating static content like you are doing right now. It will automatically detect if there's a .gz file and serve that instead when needed.

3. Use one of the Apache cache mechanisms like mod_mem_cache or mod_disk_cache to make sure that every regularly used file will be in cache. Tutorial: http://webdirect.no/linux/apache-caching-with-gzip-enabled/

4. Use a caching proxy like Varnish in front of it, these type of servers have a much smarter caching mechanism and will actually cache the files that matter most.

For your current version however, something like this (untested) should do the trick:

RewriteEngine On    
RewriteCond %{HTTP:Accept-encoding} gzip
RewriteCond %{REQUEST_FILENAME}\.gz -s
RewriteRule ^(.*)\.(html|css|js|data) $1\.$2\.gz [QSA]

# Prevent double gzip and give the correct mime-type
RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.html\.gz$ - [T=text/html,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.data\.gz$ - [T=text/plain,E=no-gzip:1,E=FORCE_GZIP]

Header set Content-Encoding gzip env=FORCE_GZIP

Sacral

Interesting alternatives you write there. While nginx and varnish probably aren't for since I have an Apache Installation already and don't wont to install all of that for two files. 1 or 3 seem to be interesting Options for me to investigate. However, just fixing the .htaccess seems to be the fastest Option for me right now. I tried your suggestion: It doesn't ask to save the file anymore, but it doesn't decompress it before displaying. I see the compressed file, lot's of characters. I tested in Firefox and Chrome. – Undermanned

When including the browsers will probably be smart enough to do it correct, but I indeed forgot to pass the encoding along. I'll update the answer :) – Sacral

I use this similar method except I use it in a mod perl script that runs before any CGI. This allows me to also fetch engine specific files if needed. i.e. /file.js -> file.jscript.js.gz for ie, file.gecko.js.gz for firefox, file.v8.js.gz for chrome, or file.nitro.js.gz for safari. Works the same with CSS, but its based off of renderer instead of js engine, trident for ie, gecko for firefox, webkit for chrome and safari. If it can't find browser specific, it uses the default file.js.gz. It also takes into account software versions as well for clients. – Vibrations

This post is helpful to me. But I had a problem. If I request the css.gz file directly, the response header contains Content-Encoding. Is there any other to resolve this. – Gogh

It should contain the content-encoding=gzip because it is. If you really don't want it you can remove the Header line though, that should remove it but I wouldn't recommend it. – Sacral

Answer 3

The accepted answer seems quite painful. Wolph's answer seems better, but still requires separate configuration for each file extension and lacks support for more advanced negotiation (q-values, status 406, TCN, etc.). Rather than implementing content negotiation yourself using mod_rewrite, you may want to consider using mod_negotiation as discussed in this question. Copying my answer from there:

Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz
<FilesMatch ".+\.tar\.gz$">
    RemoveEncoding .gz
    # Note:  Can use application/x-gzip for backwards-compatibility
    AddType application/gzip .gz
</FilesMatch>

This has the added bonus of working for all .gz files rather than only explicitly-configured ones and being easily extended for brotli or other encodings.

It does have one major drawback, since only requests for files which do not exist are negotiated a file named foo.js would make requests for /foo.js (but not /foo) return the uncompressed version. This can be avoided using François Marier's solution of renaming uncompressed files with a double extension, so foo.js is deployed as foo.js.js.

Sidelight

Answer 4

What about the solution outlined here: http://feeding.cloud.geek.nz/posts/serving-pre-compressed-files-using/. Use Apache's built-in MultiViews...

Margheritamargi

This seems much more sane than mod_rewrite rules – Fitful

Answer 5

This took me a while to figure out but the reason most people are having issues with rewriting requests to pre-compressed files is because they using REQUEST_FILENAME.

It seem REQUEST_FILENAME depending on the context and where you put you rewrite code, determines if you getting the full file-system path or a copy of REQUEST_URI.

REQUEST_FILENAME
The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as when used in virtual host context, the same value as REQUEST_URI. Depending on the value of AcceptPathInfo, the server may have only used some leading components of the REQUEST_URI to map the request to a file.

Ref: https://httpd.apache.org/docs/2.4/mod/mod_rewrite.html

To fix this I found there are 2 possible options.

One is to use the look-ahead modifier as per the docs.

If used in per-server context (i.e., before the request is mapped to the filesystem) SCRIPT_FILENAME and REQUEST_FILENAME cannot contain the full local filesystem path since the path is unknown at this stage of processing. Both variables will initially contain the value of REQUEST_URI in that case. In order to obtain the full local filesystem path of the request in per-server context, use an URL-based look-ahead %{LA-U:REQUEST_FILENAME} to determine the final value of REQUEST_FILENAME.

The other option that is not reliant on context or look-ahead I found to be using a DOCUMENT_ROOT with REQUEST_URI.

So putting it all together looks something like this.

# ==============================================================================
# Serving pre-compressed content
# ==============================================================================
# Include in a vhost to enable Serving Gzip Files
# ------------------------------------------------------------------------------
# Ref:
# - https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#precompressed
#

<IfModule mod_headers.c>
    <IfModule mod_rewrite.c>
        RewriteEngine On

        # Serve gzip compressed CSS and JS files if they exist
        # and the client accepts gzip.
        RewriteCond "%{HTTP:Accept-encoding}" "gzip"

        # REQUEST_FILENAME is only available as a full path once the requests
        # has been resolved so we need to use a look ahead or
        # DOCUMENT_ROOT+REQUEST_URI
        # RewriteCond "%{LA-U:REQUEST_FILENAME}\.gz"    "-s"
        RewriteCond "%{DOCUMENT_ROOT}%{REQUEST_URI}.gz" "-s"

        # Rewrite to gzip file
        RewriteRule "^(.*)\.(css|js)"                   "$1\.js\.gz" [QSA]

        # Serve correct content types, and prevent mod_deflate double gzip.
        RewriteRule "\.css\.gz$" "-" [T=text/css,E=no-gzip:1]
        RewriteRule "\.js\.gz$"  "-" [T=text/javascript,E=no-gzip:1]

        <FilesMatch "(\.js|\.css)$">
            Header append Pre-Compressed 0
        </FilesMatch>

        <FilesMatch "(\.js\.gz|\.css\.gz)$">
            # Serve correct encoding type.
            Header append Content-Encoding gzip

            # Force proxies to cache gzipped &
            # non-gzipped css/js files separately.
            Header append Vary Accept-Encoding
            Header append Pre-Compressed 1
        </FilesMatch>
    </IfModule>
</IfModule>

It would be nice if Apache's official docs actually point this out or used one of these 2 examples that work in all contexts as this has probably tripped up a lot of people over the years.

Ref: https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#precompressed

Yardman