Zip on-the-fly compression library in C for streaming

Question

Is there a library for creating zip files (the zip file format not gzip or any other compression format) on-the-fly (so I can start sending the file while it is compressing) for very large files (4 Gb and above).

The compression ratio does not matter much (mostly media files).

The library has to have a c-interface and work on Debian and OSX.

Statampere

@karlphillip Sorry about that. Debian and OSX support is necessary. – Statampere

Answer 1

libarchive supports any format you want, on the fly and even in-memory files.

Reeher

Seems very promising. Will check it out. – Statampere

Answer 2

zlib supports compressing by chunks. you should be able to start sending a small chunk right after compressing it, while the library is still compressing the next chunk. (see this example)

(unfortunately, the file table is stored at the end of the zip file, so the file will be unusable until it is complete on the receiver side)

Brandwein

I was under the impression that zlib cannot handle zip-files? See zlib.net/zlib_faq.html#faq11 – Statampere

Not true about zip files, the directory is added at the end to simplify random access, but the file entries themselves are written sequentially. The directory at the end is only for convenience. – Malicious

@DanielW: that same FAQ link points to contrib/minizip which does create zip files using zlib, so it obviously can. – Jannelle

@Jannelle no. The Minizip project have a zlib dependency and can read zip files. That doesn't automatically mean that Minizip has support for on-the-fly compression nor does it mean that zlib supports zip-files. If the answer was phrased as: You can create this zip library yourself easily by using zlib or you can check if Minizip has support for on-the-fly compression I would be more inclined to mark it as accepted. – Statampere

well, technically, pointing you toward a library then telling you to check for yourself is not an answer to your question... that's why i will not correct this answer. – Brandwein

also i had a look at the LZMA library hosted on 7-zip.org: it supports decompressing in chunks but not compressing... – Brandwein

now there is another answer which seems more valid than mine, i am considering deleting this answer... – Brandwein

Answer 3

While this question is old and already answered I will note a new potential solution for those that find this.

I needed something very similar, a portable and very small library that created ZIP archives in a streaming fashion in C. Not finding anything that fit the bill I created one that uses zlib, available here:

https://github.com/CTrabant/fdzipstream

That code only depends on zlib and essentially provides a simple interface to creating ZIP archives. Most importantly (for me) the output can be streamed to a pipe, socket, whatever as the output stream does not need to be seek-able. The code is very small, a single source file and a header file. Works on OSX and Linux and probably elsewhere. Hope it helps someone beyond just me...

Meli