LzmaLib: compress / decompress buffer in C

Question

I'm trying to use LzmaLib's LzmaCompress() and LzmaDecompress() with buffers, adapting the examples provided here.

I'm testing with a ~3MB buffer and the compression function seems to work fine (produces a ~1.2MB compressed buffer), but when I try to decompress, it just extracts ~300 bytes and returns SZ_ERROR_DATA.

The few extracted bytes are right, but I don't know why it stops there.

My code:

#include <stdio.h>
#include <stdlib.h>

#include "LzmaLib.h"

void compress(
    unsigned char **outBuf, size_t *dstLen,
    unsigned char *inBuf, size_t srcLen)
{
    unsigned propsSize = LZMA_PROPS_SIZE;
    *dstLen = srcLen + srcLen / 3 + 128;

    *outBuf = (unsigned char*)malloc(propsSize + *dstLen);

    int res = LzmaCompress(
        (unsigned char*)(*outBuf + LZMA_PROPS_SIZE), dstLen,
        inBuf, srcLen,
        *outBuf, &propsSize,
        -1, 0, -1, -1, -1, -1, -1);

    assert(res == SZ_OK);

    *dstLen = *dstLen + LZMA_PROPS_SIZE;
}

void uncompress(
    unsigned char **outBuf, size_t *dstLen,
    unsigned char *inBuf,  size_t srcLen
) {
    *dstLen = 5000000;
    *outBuf = (unsigned char*)malloc(*dstLen);

    srcLen = srcLen - LZMA_PROPS_SIZE;
    int res = LzmaUncompress(
        *outBuf, dstLen,
        (unsigned char*)(inBuf + LZMA_PROPS_SIZE), &srcLen,
        inBuf, LZMA_PROPS_SIZE);

    assert(res == SZ_OK);
}

void do_compress() {
    FILE* file = fopen("Module.dll", "r");
    size_t size, decSize;
    unsigned char *data, *dec = NULL;

    fseek(file, 0L, SEEK_END);
    size = ftell(file);
    fseek(file, 0L, SEEK_SET);

    data = (unsigned char*)malloc(size);
    fread(data, 1, size, file);
    fclose(file);

    compress((unsigned char**)&dec, &decSize, data, size);

    file = fopen("Module.lzma", "w");
    fwrite(dec, 1, decSize, file);
    fclose(file);
}

void do_uncompress() {
    FILE* file = fopen("Module.lzma", "r");
    size_t size, decSize;
    unsigned char *data, *dec = NULL;

    fseek(file, 0L, SEEK_END);
    size = ftell(file);
    fseek(file, 0L, SEEK_SET);

    data = (unsigned char*)malloc(size);
    fread(data, 1, size, file);
    fclose(file);

    uncompress((unsigned char**)&dec, &decSize, data, size);

    file = fopen("Module_DEC.dll", "w");
    fwrite(dec, 1, decSize, file);
    fclose(file);
}

int main()
{
    do_compress();
    do_uncompress();

    return 0;
}

If this code is not the better way to compress buffers with LzmaLib, I'm happy to accept suggestions.

Skivvy

I pass dstLen and the problem persists. Only a few bytes are decompressed and error 1 (SZ_ERROR_DATA) is returned. – Skivvy

Answer 1

I bet the problem lurks in how you read/write your files. You need to open them in binary mode to prevent any substitutions during read/write operations.

Change all instances of:

• fopen(xxx, "r") -> fopen(xxx, "rb")
• fopen(xxx, "w") -> fopen(xxx, "wb")

Stu

Man, I can't say the words I'm thinking right now. That was the exact problem. Thank you very much. – Skivvy

Little mistakes are the hardest to find :) – Stu

Answer 2

I didn't check this specificly for LzmaCompress but most of the other compressing libraries like libz handle that function similar to the standard read/write or fread/fwrite functions, i.e. allowing you to continuously calling the functions to compress more and more data in one stream. So at some point, you will have to say "I'm done, please flush everything not written so far". Possibly, you forgot that part. If not, a Minimal, Complete, and Verifiable example would be cool.

Mandamus

I understand, but LzmaCompress() and LzmaDecompress() are helper functions that compress/decompress a full block of data with a single call. – Skivvy

then the part about MCVE remains. – Mandamus

Answer 3

When you compress, you pass the number of compressed output bytes to the caller. But your buffer contains LZMA_PROPS_SIZE additional bytes. So, when writing the lzma file, you actually forget the last LZMA_PROPS_SIZE bytes and on later reading, those are missing.

Mandamus

That's true. I've added LZMA_PROPS_SIZE to the result size but the problem still persists. – Skivvy