gzip + chunked : must wait the whole file to be downloaded before unzipping?

Question

I am pretty sure of the answer but I would like someone to confirm it please.

There is no way to unzip only a part of a file when gzip is used in the HTTP headers. I gotta download the whole file before to be able to unzip it to get the data.

Right ?

For example, if I get the first 100 bytes with some code like that:

myfile.read(100)

I won't be able to unzip it at this point.

Thanks.

Hutment

Answer 1

You can start decompressing a gzip stream immediately, for whatever amount of data you have so far. You will be able to extract all of the uncompressed bytes represented in the compressed data you have available so far.

You must always decompress from the beginning. So what you can't do is start decompressing in the middle of a gzip stream. If you want to access data in the middle, you need to decompress all of the data up to that point.

Sturgis

Ok thank you. I ended up using Requests and indeed it looks like it handles it correctly so I might have done something wrong when I used urllib2. – Hutment

Answer 2

Wrong. GZIP allows streaming. You might be confusing the format with the ZIP archive format.

Cordwain