GZIP compression using zlib into buffer

Question

I want to compress a memory buffer using gzip and put the compressed bytes into another memory buffer. I want to send the compressed buffer in the payload of a HTTP packet with Content-Encoding: gzip. I can easily do this using zlib for deflate compression ( compress() function ). However, there is no API that I see for what I need ( gzip ). The zlib API is to compress and write to a file ( gzwrite() ). However, I want to compress and write to a buffer.

Any ideas?

I am in C on Linux.

Congregationalism

AFAIK, zlib's gz API needs a real FD, does open_memstream mmaps the buffer provided? in this case are you sure the FILE has a valid fd? – Rudiger

Possible duplicate of How to compress a buffer with zlib? – Sandpit

Answer 1

deflate() works in zlib format by default, to enable gzip compressing you need to use deflateInit2() to "Add 16" to windowBits as in the code below, windowBits is the key to switch to gzip format

// hope this would help  
int compressToGzip(const char* input, int inputSize, char* output, int outputSize)
{
    z_stream zs;
    zs.zalloc = Z_NULL;
    zs.zfree = Z_NULL;
    zs.opaque = Z_NULL;
    zs.avail_in = (uInt)inputSize;
    zs.next_in = (Bytef *)input;
    zs.avail_out = (uInt)outputSize;
    zs.next_out = (Bytef *)output;

    // hard to believe they don't have a macro for gzip encoding, "Add 16" is the best thing zlib can do:
    // "Add 16 to windowBits to write a simple gzip header and trailer around the compressed data instead of a zlib wrapper"
    deflateInit2(&zs, Z_DEFAULT_COMPRESSION, Z_DEFLATED, 15 | 16, 8, Z_DEFAULT_STRATEGY);
    deflate(&zs, Z_FINISH);
    deflateEnd(&zs);
    return zs.total_out;
}

Some relevant contents from their header:

"This library can optionally read and write gzip and raw deflate streams in memory as well."

"Add 16 to windowBits to write a simple gzip header and trailer around the compressed data instead of a zlib wrapper"

It's funny document of deflateInit2() is 1000+ lines away from its definition, I wouldn't ready the document again unless I have to.

Tyrannous

Welcome to StackOverflow. please explain how your code resolves the problem and put some description on how your code works. – Opt

Answer 2

No, the zlib API does in fact provide gzip compression in memory with the deflate functions. You need to actually read the documentation in zlib.h.

Pneumograph

I was initially confused where I have heard the name :D ... you are ofcourse right the "basic" functions will achieve that. – Rudiger

Answer 3

Gzip is a file format that's why it seems the utility functions provided operates on a fd, use shm_open() to create an fd mmap() with sufficient memory. It is important that the data being written doesn't extend the size of the mapped region otherwise the write will fail. That's a limitation with mmapped region.

Pass the fd to gzdopen().

But as Mark suggested in his answer using Basic API interface is a better way.

Rudiger