DeflateStream 4GB Limit in .NET

Question

From MSDN: DeflateStream Class

DeflateStream cannot be used to compress files larger than 4 GB.

Are there any other implementations for .NET without the 4 GB limit?

NOTE: I really need to decompress a file in GZ format with content larger than 4 GB. Can any code do that?

Graziano

Answer 1

Take a look at SharpZipLib. Not sure if it's subject to the same limitation, but worth a look.

Dittmer

Answer 2

FYI, we have removed the 4 GB limit from DeflateStream in .NET 4.

Hindward

But someone like me who is deflating gzipped files in SSIS is out of luck - you guys skipped releasing a new version with vs2010 / sql2008 r2 :( – Aikoail

Answer 3

There is sample code at CodeProject using the 7-Zip library.

The license is open, so you should be able to use this in your project.

7-Zip also supports GZ files.

Suppuration

Would up vote this if I had some votes left, this seems like a good one and looking at the change log does have Zip64 support. – Apomorphine

Way better than SharpZipLib (which is a facade for zlib). 7zip is amazing. – Gladiate

Answer 4

Having a look around, it seems a lot of people have encountered this problem. System.IO.Compressio.DeflateStream clarifications please seems to be the most comprehensive.

The only implementation I was able to find that seems to overcome this problem by using Zip64 is Xceed Zip for .NET.

However, it is very expensive and I am not sure if it would suite your needs.

Edit:

There does seem to be quite a number of implementations of Zip64 for .NET, but I can't find any that are free.

Apomorphine

That external thread is not helpful. It is filled with supposition, and wrong ones. The DeflateStream does not keep all the stream data in memory - that is not the reason for the 4gb limitation. – Hydrostat

DotNetZip is a managed .NET library for ZIP files, that does ZIP64. It's free (gratis + libre), though donations are encouraged. ZIP64 is a solution to a different problem than the original poster asked about - the limitation of the BCL DeflateStream. – Hydrostat

Answer 5

Look for libraries that support DEFLATE64 (not Zip64, that's an extension to the ZIP file format). Xceed Zip for .NET does support Deflate64, and I'm sure others do too.

Kuhlmann

Um, no. deflate64 has nothing whatsoever to do with the size of the data. It only differs from deflate in providing a 64K sliding window instead of a 32K sliding window. – Camisado

Answer 6

DotNetZip does ZIP64 for .NET, and it is free. But Zip64 is not the same as Deflate64.

Hydrostat

Answer 7

Although that documentation says the 4GB limitation is for both the DeflateStream and GZipStream, only GZipStream is limited because of the CRC32 checksum. If you do not need CRC32 then use DeflateStream.

Regulus

Interesting. Couple questions. #1: Are you sure about this? #2. Why does the CRC32 limit the size of the data that can be compressed? #3. What is the source of your information? (are you familiar with the DeflateStream implementation, or what?) – Hydrostat

Hi, 1. I am absolutely sure - I have tested it. 2. See Wikipedia: en.wikipedia.org/wiki/Cyclic_redundancy_check 3. I used .NET Reflector (red-gate.com/products/reflector) to peek at the code. The 4GB limitation is completely ignored if CRC32 is not used. – Regulus