How can I pre-compress files with mod_deflate in Apache 2.x?

Question

I am serving all content through apache with Content-Encoding: zip but that compresses on the fly. A good amount of my content is static files on the disk. I want to gzip the files beforehand rather than compressing them every time they are requested.

This is something that, I believe, mod_gzip did in Apache 1.x automatically, but just having the file with .gz next to it. That's no longer the case with mod_deflate.

Halfcocked

I don't think you're going to save much though; with modern web servers, the cost of compressing the content on the fly is negligible. – Ladew

I'm running the web server in a Xen VM so I'd like to conserve as much CPU as possible for the other VMs. Also I was able to double the request rate measured with httperf on a pre-compressed 55k file compared to compressing on the fly. – Halfcocked

See #9077252 – Crossroad

Answer 1

This functionality was misplaced in mod_gzip anyway. In Apache 2.x, you do that with content negotiation. Specifically, you need to enable MultiViews with the Options directive and you need to specify your encoding types with the AddEncoding directive.

Carlos

#9077252 – Crossroad

Answer 2

To answer my own question with the really simple line I was missing in my confiuration:

Options FollowSymLinks MultiViews

I was missing the MultiViews option. It's there in the Ubuntu default web server configuration, so don't be like me and drop it off.

Also I wrote a quick Rake task to compress all the files.

namespace :static do
    desc "Gzip compress the static content so Apache doesn't need to do it on-the-fly."
    task :compress do
        puts "Gzipping js, html and css files."
        Dir.glob("#{RAILS_ROOT}/public/**/*.{js,html,css}") do |file|
            system "gzip -c -9 #{file} > #{file}.gz"
        end
    end
end

Halfcocked

Answer 3

It is possible to serve pre-compressed files using mod_negotiation although it is a bit finicky. The primary difficulty is that only requests for files which do not exist are negotiated. So if foo.js and foo.js.gz both exist, responses for /foo.js will always be uncompressed (although responses for /foo would work correctly).

The easiest solution I've found (from François Marier) is to rename uncompressed files with a double file extension, so foo.js is deployed as foo.js.js so requests for /foo.js negotiate between foo.js.js (no encoding) and foo.js.gz (gzip encoding).

I combine that trick with the following configuration:

Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz

# Send .tar.gz without Content-Encoding: gzip
<FilesMatch ".+\.tar\.gz$">
    RemoveEncoding .gz
    # Note:  Can use application/x-gzip for backwards-compatibility
    AddType application/gzip .gz
</FilesMatch>

I wrote a post which discusses the reasoning for this configuration and some alternatives in detail.

Gastroenterostomy

This doesn't seem to work any more. With Apache 2.4.25 I always get uncompressed results, probably becuase the MultiViews only kicks in if the file doesn't exist as named. The doc says as much. Pity! – Hoplite

Good catch @MvG, you are absolutely right! I've updated the answer with a workaround, although it is not ideal as it requires renaming the uncompressed file and a bit of configuration trickery. Hope it helps! – Gastroenterostomy

@Kevinoid: That's exactly the workaround I came up with myself, after sleeping on it. Nice! – Hoplite

Answer 4

I am afraid MultiViews will not work as expected: the doc says Multiviews works "if the server receives a request for /some/dir/foo, if /some/dir has MultiViews enabled, and /some/dir/foo does not exist...", in other words: if you have a file foo.js and foo.js.gz in the same directory, just activating MultiViews will not cause the .gz file to be sent even if the AcceptEncoding gzip header is transmitted by the browser (you can verify this behavior by temporarily disabling mod_deflate and monitoring the response with e.g. HTTPFox).

I am not sure if there is a way around this with MultiViews (maybe you can rename the original file and then add a special AddEncoding directive), but I believe you can construct a mod_rewrite rule to handle this.

Numidia

I'm sure it worked at the time, and I'm sure I didn't delete the original files. It's plausible mod_rewrite was involved, somehow. I probably had a rule that was "serve this statically if the file exists". I may have changed it to include the .gz extension, but I no longer have access to that system to verify. – Halfcocked

If in your example you request just foo rather than foo.js, it will work. If the client accepts gzip they will get foo.js.gz else they will get foo.js. – Oteliaotero

see #9077252 – Crossroad

Answer 5

I have an Apache 2 built from source, and I found I had to modify the following in my httpd.conf file:

Add MultiViews to Options:

Options Indexes FollowSymLinks MultiViews

Uncomment AddEncoding:

AddEncoding x-compress .Z
AddEncoding x-gzip .gz .tgz

Comment AddType:

#AddType application/x-compress .Z
#AddType application/x-gzip .gz .tgz

Extravasation

Answer 6

mod_gzip compressed content on the fly as well. You can pre-compress the files by actually logging into your server, and doing it from shell.

cd /var/www/.../data/
for file in *; do
    gzip -c $file > $file.gz;
done;

Ladew

This will remove the original files, which means clients that don't have Aceept-Encoding: gzip won't be serviced. – Halfcocked

While you're editing, why not add -9 and get the highest compression possible. My 1500 files compressed in 38 seconds, so it's worth doing to save every byte possible in bandwidth and download time. :) (Also wishing I could edit my typo in my previous comment. Ugh) – Halfcocked

Not according to the man page on my Mac, it says -6 is the default. – Halfcocked

Answer 7

You can use mod_cache to proxy local content in memory or on disk. I don't know if this will work as expected with mod_deflate.

Orotund

Answer 8

I have a lot of big .json files. Most readers are in this situation. The preview answers didn't talk about the returned "Content-type".

I you want the following request return a pre-compressed file with "Content-Type: application/json" transparently, use Multiview with ForceType

http://www.domain.com/(...)/bigfile.json
-> Content-Encoding:gzip, Content-Type: Content-Encoding:gzip

1) files must be rename: "file.ext.ext"

2) Multiview works great with ForceType

In the file system:

// Note there is no bigfile.json
(...)/bigfile.json.gz
(...)/bigfile.json.json

In your apache config:

<Directory (...)>
    AddEncoding gzip .gz
    Options +Multiviews
    <Files *.json.gz>
        ForceType application/json
    </Files>
</Directory>

Short and simple :)

Vella