What's the best way to implement file compression?

Question

Edit: Clarification after answer: I chose Lucas's answer as the correct answer for this question. DotNetZip, when implemented with the I18N.dlls works like a champ in the Editor and built .exes. My specific implementation details are located in the "Potential Answer: DotNetZip" answer below.*

Edit 1/30/2011: Just a quick note to say that I have been using the DotNetZip solution in 3.x without any issues.

Using the System.IO.Compression.GZipStream classes results in the dreaded DllNotFoundException: MonoPosixHelper exception (http://answers.unity3d.com/questions/3147/dllnotfoundexception-monoposixhelpersystem-io-compression-deflatestream), so until Unity 3.0 comes out and the later version of Mono it ships with supports a fully managed compression scheme, we have to rely on a third party compression library.

There seem to be two libraries that might fit the bill. SharpZipLib (http://www.icsharpcode.net/opensource/sharpziplib/) is a venerable tool for zip compression, but the API is painful to use.

Some have reported that DotNetZip (http://dotnetzip.codeplex.com) works with Unity. I keep receiving an "IBM437 codepage not supported" (or similar) error when trying to work with this library. It is a much better library than SharpZipLib from a usability perspective.

There is a mono compiled version of SharpZipLib that ships with Unity (as discussed in this forum post http://forum.unity3d.com/viewtopic.php?t=10699&highlight=sharpziplib), which seems like a possibility. However, I am hesitant to use an older version of the library.

Should we just wait and hope? Can someone confirm if later versions of Mono will support this out of the box? Or will we have to roll our own in the 3.0 time frame, too?

Tendentious

Has anyone tried using this for iOS? I'm concerned firstly whether it will work and secondly about speed!

– Anagram

Answer 1

Nobody said that Unity3 will ship with a mono that has a fully managed GZipStream implementation. In fact, being the guy that is working on this currently, I can tell you that that would be extremely unlikely. So yes, also for the Unity3.0 timeframe, you'll have to rely on a managed implementation.

I went and tried the 3rd party library you are having trouble with: DotNetZip. I downloaded their distribution package, and grabbed Ionic.Zip.dll from it, and placed it into the Assets folder of an empty Unity2.6.1 project.

I then wrote a quick testscript:

using UnityEngine;
using Ionic.Zip;

public class NewBehaviourScript : MonoBehaviour {
    // Use this for initialization
    void Start () {
        using (ZipFile zip = new ZipFile())
        {
             // add this map file into the "images" directory in the zip archive
             zip.AddFile("c:\\mono.patch", "images");
             // add the report into a different directory in the archive
             zip.Save("MyZipFile.zip");
        }
    }
}

(copied almost verbatim from their website). c:\mono.patch is a file that happened to live in my c:\ drive. When I ran this script, it succesfully wrote MyZipFile.zip.

Naturally the .AddFile() won't work from a webplayer, but this library has entrypoints for creating new files in the zipfile from a stream as well.

If this does not work for you, please describe the exact steps you're taking, and the exact error you get.

Khania

I wish it were this easy, Lucas. But the behavior when reading from a zipfile differs between running in the editor and running a built exe. When run in the editor, unity is able to retrieve data from the zipfile, but when run in a built binary the following exception is generated: ArgumentException: Encoding name 'IBM437' not supported Parameter name: name Demeter.DataArchive.GetItemFromArchive[StringLibrary] (System.String folder, System.String fileName) [0x00000] Demeter.DataArchive.GetStringLibrary (System.String filename, StringLibraryFields language) < snip />

– Tendentious

grr... i'll post an answer with the detail

– Tendentious

Lucas? any progress?

– Tendentious

Try grabbing the I18N*.dll assemblies from the mono framework that unity ships, and put those next to "mscorlib.dll" in the built exe. Good chance that would make it work.

– Khania

I can confirm that the solution proposed by Lucas fixes this problem (tested in Unity 3.0 b7, but presumably works in older versions, too). Though, actually, all you have to do is drop them into Assets/Plugins in your Unity project to make this work. On windows, the source files needed were in C:\Program Files (x86)\Unity\Editor\Data\Mono\lib\mono\unity\ The mdb files are NOT needed, you can skip those. Using DotNetZip with these extra dll files included seems to be the best solution all around by far.

– Parenthood

Answer 2

The Answer: DotNetZip

I know it's bad juju to use an answer to post a comment, but I need to show some detail. Please accept my apologies.

Here are the implementation details that were answered by Lucas and confirmed by x4000:

Using DotNetZip version 1.9, place the Ionic.Zip.reduced.dll in your Assets folder.

Then, from your (Windows) C:\Program Files (x86)\Unity\Editor\Data\Mono\lib\mono\unity folder, copy the I18N*.dll files to your Assets folder. There should be 6. These files will prevent the IBM487 errors.

My methods for reading and writing to data files are:

    /// <summary>
    /// Puts an item in the archive, overwrites if already exists
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="folder"></param>
    /// <param name="fileName"></param>
    /// <param name="item"></param>
    public void AddItemToArchive<T>(string folder, string fileName, ref T item)
    {
        if (!item.GetType().IsSerializable)
            throw new ArgumentException("item must be serializable");

        using (ZipFile zipFile = new ZipFile())
        {
            zipFile.UseUnicodeAsNecessary = true;

            //serialize item to memorystream
            using (MemoryStream m = new MemoryStream())
            {
                IFormatter formatter = new BinaryFormatter();
                formatter.Serialize(m, item);
                m.Position = 0;

                ZipEntry zipEntry = zipFile.AddEntry("entry", m);
                zipEntry.UseUnicodeAsNecessary = true;
                zipEntry.Password = _password;

                zipFile.Save(filePath);
            }
        }
    }

    /// <summary>
    /// Retrieves an item from the archive
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="folder"></param>
    /// <param name="fileName"></param>
    /// <returns></returns>
    public T GetItemFromArchive<T>(string folder, string fileName)
    {
        //get the stream from the archive
        using (MemoryStream m = new MemoryStream())
        {
            using (ZipFile zipFile = new ZipFile(filePath))
            {
                zipFile.UseUnicodeAsNecessary = true;

                ZipEntry e = zipFile["entry"];
                e.UseUnicodeAsNecessary = true;
                e.Password = _password;
                e.Extract(m);
                m.Position = 0;

                //now serialize it back to the correct type
                IFormatter formatter = new BinaryFormatter();
                T item = (T)formatter.Deserialize(m);

                return item;
            }
        }
    }

I hope this helps. ;o)

Tendentious

Just FYI, you can edit your original answer if you need to clarify anything. ;)

– Reversion

Did not work with iOS, I'm still getting IBM437 errors. :(

– Shiva

Answer 3

Have you tried looking at 7Zip? Has a completely managed C# library that I know works in Unity. It gives you compress/uncompress to and from memory streams/byte arrays, in LZMA format, so it's better than zip and actually what Unity uses to compress assets.

http://www.7-zip.org/sdk.html

If you need something compatible with .zip format you might have some trouble.

Barbusse

holy macaroni, if you thought SharpZipLib was painful to use, you need to look at the 7zip libs. Fake COM calls, zero documentation, cryptic code... Just terrible. There are several wrappers for this on CodePlex, but they hide this confusion with extra dependencies. Mostly dealing with the original problem with the 7zip dll's. 64/32 bit dependency issues, lack of implemented features (streams or encryption, etc). Are there any samples for use in Unity?

– Tendentious

Answer 4

Potential Answer: SharpZipLib

There has been no progress on this question so I will post my research into this potential answer.

http://www.icsharpcode.net/opensource/sharpziplib/

The current version of SharpZipLib seems to work. Code will compile and run in the Unity editor. But the same problem with DotNetZip arises:

NotSupportedException: CodePage 437 not supported

is thrown when the project is built and running on the mono bits.

Another SharpZipLib/Unity developer banged his head on the wall last year and was able to get the code to work with the Unity version of the SharpZipLib library: http://community.sharpdevelop.net/forums/p/9762/27065.aspx

His bug demo code was still posted, so I grabbed the .dll from it and used it in my code. It WORKS! But, I won't immediately call this an acceptable answer to this question. This is the version of the library that ships with Unity (located here: Unity\Editor\Data\Frameworks\Mono.framework), so whatever bugfixes or improvements may have happened after this version was published, or may happen in the future, are lost to me.

This is the closest answer yet, but we're still not there yet. If others post their experiences with this solution, then I may feel better about calling this the answer.

Tendentious

Answer 5

CodePage 437 / IBM437 is an error thrown by the Text.Encoding end, which is not implemented in mono 1.2.5

What you might give a look at is http://cheeso.members.winisp.net/DotNetZipHelp/html/2418e26f-685a-b5b8-5c79-984d5b8da61a.htm, a documentation page in relation to the DotNetZip lib (Ionic Zip) which talks about the fact. Going from that, the issue is just a setup problem with you not telling it to use UTF-8 so it use IBM437 which will not work

Ceramic

Thanks, Dreamora. I had used ZipFile constructor to force UTF8 encoding. This time I used the UseUnicodeAsNecessary property, and it still throws the "ArgumentException: Encoding name 'IBM437' not supported" exception. Because this happens with both DotNetZip and SharpZipLib, I believe this to be a Mono bug with encoding. The only potential solution I can see right now is to recompile the library in Mono. Unity ships with an old version of SharpZipLib compiled for Mono (don't know why or what version) that works. This may be the least painful solution.

– Tendentious

This may also be a Unity bug as the behavior differs between the editor and in built code. I'll leave that investigation for Lucas... ;o)

– Tendentious

Answer 6

The DotNetZip file also ships with an Ionic.Zlib.dll, dropping this into your project's root asset directory and then replacing any System.IO.Compression references to Ionic.Zlib will get you access to the GZipStream class, if your intention is to use memory streams rather than disk files.

Barbusse

Answer 7

I’ve been trying to extract a zip to a stream. It works great in the editor. The webplayer doesn’t seems to work though. Are there known issues with the webplayer when using the DotNetZip implementation?

Bequest

Answer 8

Bit of an update -

Both Ionic.Zlib and Ionic.Gzip work great on Unity 3.5.3 for Window / mac editor. This does not include the

I began to have issues when I compiled for the I18N*.dll assemblies discussed above.

When I compiled to the iPhone, the error mentioned at the top occurred: “IBM437 codepage not supported”

I attempted to bring all the I18N*.dll’s over, and I got the error that I18N.Rare.dll could not be cross compiled with AOT.

I then deleted that file and attempted again - and everything worked great.

So as for Unity 3.5.3 - You could bring in all I18N* files, but do not include the I18N.Rare.dll.

Thanks for this post - it was very helpful.

Diminutive

how did you integrate the I18N*.dll to work with iOS? i dragged the dlls to the Assets/Plugins. Do i need to do extra Xcode work?

– Throstle

Answer 9

Thanks for this post, was very useful. I would like to add that on iOS webrequests are automatically uncompressed when setting the requestheader to “gzip”, so on iOS is no need for an extra uncompression library.

Eggers