msgpack C++ implementation: How to pack binary data?

Question

I am making use of C++ msgpack implementation. I have hit a roadblock as to how to pack binary data. In terms of binary data I have a buffer of the following type:

unsigned char* data;

The data variable points to an array which is actually an image. What I want to do is pack this using msgpack. There seems to be no example of how to actually pack binary data. From the format specification raw bytes are supported, but I am not sure how to make use of the functionality.

I tried using a vector of character pointers like the following:

msgpack::sbuffer temp_sbuffer;
std::vector<char*> vec;
msgpack::pack(temp_sbuffer, vec);

But this results in a compiler error since there is no function template for T=std::vector.

I have also simply tried the following:

msgpack::pack(temp_sbuffer, "Hello");

But this also results in a compilation error (i.e. no function template for T=const char [6]

Thus, I was hoping someone could give me advice on how to use msgpack C++ to pack binary data represented as a char array.

Fame

Answer 1

Josh provided a good answer but it requires the copying of byte buffers to a vector of char. I would rather minimize copying and use the buffer directly (if possible). The following is an alternative solution:

Looking through the source code and trying to determine how different data types are packed according to the specification I happened upon msgpack::packer<>::pack_raw(size_t l) and msgpack::packer<>::pack_raw_body(const char* b, size_t l). While there appears to be no documentation for these methods this is how I would described them.

1. msgpack::packer<>::pack_raw(size_t l): This method appends the type identification to buffer (i.e. fix raw, raw16 or raw32) as well as the size information (which is an argument for the method).
2. msgpack::packer<>::pack_raw_body(const char* b, size_t l): This method appends the raw data to the buffer.

The following is a simple example of how to pack a character array:

msgpack::sbuffer temp_sbuffer;
msgpack::packer<msgpack::sbuffer> packer(&temp_sbuffer);
packer.pack_raw(5);  // Indicate that you are packing 5 raw bytes
packer.pack_raw_body("Hello", 5); // Pack the 5 bytes

The above example can be extended to pack any binary data. This allows one to pack directly from byte arrays/buffers without having to copy to an intermediate (i.e. a vector of char).

Fame

What would the corresponding unpacking look like? – Robespierre

Answer 2

If you can store your image in a vector<unsigned char> instead of a raw array of unsigned char, then you can pack that vector:

#include <iostream>
#include <string>
#include <vector>
#include <msgpack.hpp>

int main()
{
    std::vector<unsigned char> data;
    for (unsigned i = 0; i < 10; ++i)
        data.push_back(i * 2);

    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, data);

    msgpack::unpacked msg;
    msgpack::unpack(&msg, sbuf.data(), sbuf.size());

    msgpack::object obj = msg.get();
    std::cout << obj << std::endl;
}

Strangely, this only works for unsigned char. If you try to pack a buffer of char instead (or even an individual char), it won't compile.

Lely

Thanks Josh. I thought I had tried this already. Unfortunately with this solution I have to copy a lot of binary data (images) from byte arrays to vectors of unsigned char. As well, I am not entirely sure why yours compiles. As I don't see any functions which specifically support unsigned char. I am going to dig deeper into the source code and see if there is a more efficient way I can get this done (i.e. avoiding copying). – Fame

Also, this will save it as an array of chars, instead of binary data, which is a different type according to the implementation... – Valuation

Just looked it up: using a vector<char> instead of vector<unsigned char> seems to use the binary type instead of the array... – Valuation

Answer 3

MessagePack has a raw_ref type which you could use like so:

#include "msgpack.hpp"

class myClass
{
public:
    msgpack::type::raw_ref r;
    MSGPACK_DEFINE(r);
};

int _tmain(int argc, _TCHAR* argv[])
{
    const char* str = "hello";

    myClass c;

    c.r.ptr = str;
    c.r.size = 6;

    // From here on down its just the standard MessagePack example...

    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, c);

    msgpack::unpacked msg;
    msgpack::unpack(&msg, sbuf.data(), sbuf.size());

    msgpack::object o = msg.get();

    myClass d;
    o.convert(&d);

    OutputDebugStringA(d.r.ptr);

    return 0;

}

Disclaimer: I found this by poking around the header files, not through reading the non-existent documentation on serialising raw bytes, so it may not be the 'correct' way (though it was defined along with all the other 'standard' types a serialiser would want to explicitly handle).

Lourielouse

I have just confirmed that this works for me, on encoding at least. I populate a std::vector<char>, point a raw_ref into the vector, and pack the raw_ref. The result uses the msgpack type code C4, which is the ID for the "bin 8" format. – Tradelast

On unpacking, there is a catch to this: The library will unpack a character stream into a raw_ref well enough, but then it isn't clear, to an ignorant observer, what owns the memory pointed to by the raw_ref. My casual analysis suggests it's the msgpack::unpacked structure; if that's right then I suppose one must keep that structure around after unpacking for as long as the unpacked raw_ref may need to be read. ...But maybe that's already true for other unpacked types as well? I'm not sure. This library really does need better documentation. :( – Tradelast

Answer 4

msgpack-c has been updated after question and answers were posted. I'd like to inform the current situation.

Since msgpack-c version 2.0.0 C-style array has been supported. See https://github.com/msgpack/msgpack-c/releases

msgpack-c can pack const char array such as "hello". Types conversion rule is documented https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor#predefined-adaptors.

char array is mapped to STR. If you want to use BIN instead of STR, you need to wrap with msgpack::type::raw_ref. That is packing overview.

Here are unpacking and converting description: https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_object#conversion

Unpack means creating msgpack::object from MessagePack formatted byte stream. Convert means converting to C++ object from msgpack::object.

If MessagePack formatted data is STR, and covert target type is char array, copy data to array, and if array has extra capacity, add '\0'. If MessagePack formatted data is BIN, '\0' is not added.

Here is a code example based on the original question:

#include <msgpack.hpp>
#include <iostream>

inline
std::ostream& hex_dump(std::ostream& o, char const* p, std::size_t size ) {
    o << std::hex << std::setw(2) << std::setfill('0');
    while(size--) o << (static_cast<int>(*p++) & 0xff) << ' ';
    return o;
}

int main() {
    {
        msgpack::sbuffer temp_sbuffer;
        // since 2.0.0 char[] is supported.
        // See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor#predefined-adaptors
        msgpack::pack(temp_sbuffer, "hello");
        hex_dump(std::cout, temp_sbuffer.data(), temp_sbuffer.size()) << std::endl;

        // packed as STR See https://github.com/msgpack/msgpack/blob/master/spec.md
        // '\0' is not packed
        auto oh = msgpack::unpack(temp_sbuffer.data(), temp_sbuffer.size());
        static_assert(sizeof("hello") == 6, "");
        char converted[6];
        converted[5] = 'x'; // to check overwriting, put NOT '\0'.
        // '\0' is automatically added if char-array has enought size and MessagePack format is STR
        oh.get().convert(converted); 
        std::cout << converted << std::endl;
    }
    {
        msgpack::sbuffer temp_sbuffer;
        // since 2.0.0 char[] is supported.
        // See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor#predefined-adaptors
        // packed as BIN
        msgpack::pack(temp_sbuffer, msgpack::type::raw_ref("hello", 5));
        hex_dump(std::cout, temp_sbuffer.data(), temp_sbuffer.size()) << std::endl;

        auto oh = msgpack::unpack(temp_sbuffer.data(), temp_sbuffer.size());
        static_assert(sizeof("hello") == 6, "");
        char converted[7];
        converted[5] = 'x';
        converted[6] = '\0';
        // only first 5 bytes are written if MessagePack format is BIN
        oh.get().convert(converted);
        std::cout << converted << std::endl;
    }
}

Running Demo: https://wandbox.org/permlink/mYJyYycfsQIwsekY

Backstop