gzip compression of chunked encoding response?

Question

I'm trying to get my webserver to correctly gzip an http response that is chunk encoding.

my understanding of the non-gzip response is that it looks like this:

<the response headers>

and then for each chunk,

<chunk length in hex>\r\n<chunk>\r\n

and finally, a zero length chunk:

0\r\n\r\n

I've tried to get gzip compression working and I could use some help figuring out what should actually be returned. This documentation implies that the entire response should be gzipped, as opposed to gzipping each chunk:

HTTP servers sometimes use compression (gzip) or deflate methods to optimize transmission.
Chunked transfer encoding can be used to delimit parts of the compressed object.
In this case the chunks are not individually compressed. Instead, the complete payload 
is compressed and the output of the compression process is chunk encoded.

I tried to gzip the entire thing and return the response even without chunked, and it didn't work. I tried setting the Content-Encoding header to "gzip". Can someone explain what changes must be made to the above scheme to support gzipping of chunks? Thanks.

Disfeature

Answer 1

You gzip the content, and only then apply the chunked encoding:

"Since "chunked" is the only transfer-coding required to be understood by HTTP/1.1 recipients, it plays a crucial role in delimiting messages on a persistent connection. Whenever a transfer-coding is applied to a payload body in a request, the final transfer-coding applied MUST be "chunked". If a transfer-coding is applied to a response payload body, then either the final transfer-coding applied MUST be "chunked" or the message MUST be terminated by closing the connection. When the "chunked" transfer-coding is used, it MUST be the last transfer-coding applied to form the message-body. The "chunked" transfer-coding MUST NOT be applied more than once in a message-body."

(HTTPbis Part1, Section 6.2.1)

Benedictine

the first sentence in this is really confusing. I think it should be changed.....you gzip each chunk's body (I just implemented it and it worked like a charm). you don't gzip the entire content then apply chunked encoding. you gzip each body and mark the chunk size as the compressed byte array size. – Boletus

HTTP/1.1 has defined three ways to differentiate where one message ends and the next starts(HTTP pipelining, persistent connections): content-length header, transfer-encoding header and the very least priority if none of the above are present is the closing of the connection. – Pejoration

Answer 2

In case the other answers weren't clear enough:

First you gzip the body with zlib (this can be done in a stream so you don't need the whole thing in memory at once, which is the whole point of chunking).

Then you send that compressed body in chunks (presumably the ones provided by the gzip stream, with the chunk header to declare how long it is), with the Content-Encoding: gzip and Transfer-Encoding: chunked headers (and no Content-Length header).

If you're using gzip or zcat or some such utility for the compression, it probably won't work. Needs to be zlib. If you're creating the chunks and then compressing them, that definitely won't work. If you think you're doing this right and it's not working, you might try taking a packet trace and asking questions based on that and any error messages you're getting.

Allare

FYI. convert string to gzip – Continuation

Answer 3

Likely you are not really sending an appropriately gzipped response.

Try setting the window bits to 31 in zlib. And use deflateInit2().

Alyss

This makes some assumptions about implementation. What if, to return a compressed file, he just used the "gzip" command to zip the file and returned that while setting the Content-Encoding: gzip header? Does that have the same output? – Allare

The HTTP standard really necessitates that the zlib library be used. The gzip command will output more headers but that SHOULD be compatible with zlib which is used in most other HTTP clients/servers. – Alyss

It's more likely that the poster is not setting the Content-Length header or is sending the Chunked response improperly. or is forgetting something basic like a trailing newline after the headers. – Alyss

If he's sending chunked then there shouldn't be a Content-Length header. Of course in the case where he sent it non-chunked and it "didn't work" then a Content-Length would be needed. The OP could be more helpful about what problems he's having, as his understanding at least seems correct. Given that the question was in March, he probably got it working or gave up long ago. I had to actually implement this recently, and I don't think gzipping the file with gzip and transmitting it worked. Your advice about setting window bits to 31 in zlib was very helpful, thanks. – Allare