Why do real-world servers prefer gzip over deflate encoding?

Question

We already know deflate encoding is a winner over gzip with respect to speed of encoding, decoding and compression size.

So why do no large sites (that I can find) send it (when I use a browser that accepts it)?

Yahoo claims deflate is "less effective". Why?

I maintain HTTP server software that prefers deflate, so I'd like to know if there's some really good reason not to continue doing so.

Elstan

Answer 1

There is some confusion about the naming between the specifications and the HTTP:

• DEFLATE as defined by RFC 1951 is a compressed data format.
• ZLIB as defined by RFC 1950 is a compressed data format that uses the DEFLATE data format.
• GZIP as defined by RFC 1952 is a file format that uses the DEFLATE compressed data format.

But the HTTP uses a different naming:

• gzip An encoding format produced by the file compression program "gzip" (GNU zip) as described in RFC 1952 [25]. This format is a Lempel-Ziv coding (LZ77) with a 32 bit CRC.

• deflate The "zlib" format defined in RFC 1950 [31] in combination with the "deflate" compression mechanism described in RFC 1951 [29].

So to sum up:

• gzip is the GZIP file format.
• deflate is actually the ZLIB data format. (But some clients do also accept the actual DEFLATE data format for deflate.)

See also this answer on the question What's the difference between the "gzip" and "deflate" HTTP 1.1 encodings?:

What's the difference between the "gzip" and "deflate" HTTP 1.1 encodings?

"gzip" is the gzip format, and "deflate" is the zlib format. They should probably have called the second one "zlib" instead to avoid confusion with the raw deflate compressed data format. While the HTTP 1.1 RFC 2616 correctly points to the zlib specification in RFC 1950 for the "deflate" transfer encoding, there have been reports of servers and browsers that incorrectly produce or expect raw deflate data per the deflate specification in RFC 1951, most notably Microsoft. So even though the "deflate" transfer encoding using the zlib format would be the more efficient approach (and in fact exactly what the zlib format was designed for), using the "gzip" transfer encoding is probably more reliable due to an unfortunate choice of name on the part of the HTTP 1.1 authors.

Floydflss

Answer 2

From my minimal testing it appears most HTTPds either:

1. don't support deflate on-the-fly: Apache's mod_deflate (a suprise), GWS
2. or prefer to send gzip: IIS, lighttpd's mod_compress

So to send deflate on the most popular server (Apache), you must maintain pre-encoded files and use mod_negotiate (you might even have to use type-maps to prefer deflate).

I'd guess, due to this hassle, deflate is just rarely used, and therefore bugs are more likely to exist in client deflate support than in gzip support.

Elstan

And to add to it, I read that in some cases servers send encoding labeled "Deflate" and it is really gzipped, or vice versa, and clients have adapted to deal with that bug. – Nelle

Answer 3

Check this website for more information: http://web.archive.org/web/20120321182910/http://www.vervestudios.co/projects/compression-tests

---

Deflate, per spec, is actually zlib (a compression format developed specifically for streaming content over the web)...which is a wrapper around deflate.

Internet Explorer, however, incorrectly implements HTTP 1.1 deflate (zlib) as raw deflate. So if your server sends correct HTTP 1.1 deflate (zlib) content to IE it chokes.

I've researched the topic a bit and it looks safe to ALWAYS send raw deflate to modern browsers...just make sure its is, in fact, raw and not zlib.

Check this article for more information > Gzip vs Deflate (zlib) revisited.

So I think that there is a good reason TO continue sending deflate over gzip.

Cushiony

Also see Zoompf's article referred in your SO thread here zoompf.com/blog/2012/02/lose-the-wait-http-compression with issues with (raw) Deflate mentioned. – Hartmann

My recommendation is now gzip, as many browsers have dropped raw deflate support since 2010. – Cushiony

Answer 4

As far as I know (disclaimer: and I'm not an expert here, just what I've heard), gzip uses the same algorithm as deflate but it has more header stuff that make it have a larger size (relative to deflate). However, I think deflate is supported by less clients and proxies.

Clementinaclementine

This is consistent with my knowledge as well. The difference should not be larger than a few tens of bytes... – Heavy

@ephemient: Yeah, from the size perspective, it shouldn't be a lot of difference but it seems that those bytes are checksums that need clock cycles to be generated and checked that makes gzip slower than deflate too. – Clementinaclementine

As of 6/08 fewer bots accepted it: computec.ch/projekte/browserrecon/… Some I recognize: Apache Lucene Nutch, Google Search Appliance Crawler (not Googlebot), some versions of Yahoo! Slurp (as of June 08). But any proxy/client no supporting deflate won't list it in Accept-Encoding, so there should be no harm done by simply preferring to send deflate when given the choice. I might bring this up on an Apache dev list. – Elstan

Mehrdad is correct. But none of us needs to rely on what we have heard. The spec is small and available. ietf.org/rfc/rfc1952.txt You can see for yourself that GZIP is just DEFLATE with header bytes. Ok, not quite true. In the metadata ("headers"), GZIP allows the specification of different compression methods, but defines only ONE value in the spec: that for DEFLATE. For size, the GZIP header is at least 10 bytes. In addition there are optional pieces like filename, comment and CRC16, which are in practice, almost never used in streaming scenarios. – Nelle

Answer 5

I wondered the same thing :). I think it might be to do with compatibility of older (possibly ancient) browsers. I read somewhere that older browsers are more likely to creep out at deflated content that mod_gzipped in certain instances(?) but googling this led me to conclude that it's probably best to stop googling it.

Winkler

Answer 6

ActionScript 3 has native deflate support, but for gzip you need to use an external library

Retinoscope