Serializing OpenCV Mat_<Vec3f>

Question

I'm working on a robotics research project where I need to serialize 2D matrices of 3D points: basically each pixel is a 3-vector of floats. These pixels are saved in an OpenCV matrix, and they need to be sent over inter-process communication and saved into files to be processed on multiple computers. I'd like to serialize them in an endian/architecture-independent, space-efficient way, as quickly as possible. cv::imencode here would be perfect, except that it only works on 8-bit and 16-bit elements, and we don't want to lose any precision. The files don't need to be human-readable (although we do that now to ensure data portability, and it's incredibly slow). Are there best practices for this, or elegant ways to do it?

Thanks!

Reagan

I think it's not possible to have an endian-independent file format. You could have an "indicator" integer at the beginning of the file though (something like 0xFFFE) The position of the FF and FE will tell you the order to read the file. – Dwinnell

FYI if anyone comes across this post, our group is now using ROS and its built-in serialization for OpenCV matrices. It's incredibly performant, and can log uncompressed 640x480 at over 30fps on our hardware - more importantly, it's better tested than a roll-our-own solution would ever be. But the solutions presented here are awesome nonetheless! – Reagan

Answer 1

Edit: Christoph Heindl has commented on this post with a link to his blog where he has improved on this serialisation code. Highly recommended!

http://cheind.wordpress.com/2011/12/06/serialization-of-cvmat-objects-using-boost/

--

For whoever it may benefit: Some code to serialize Mat& with boost::serialization
I haven't tested with multi-channel data, but everything should work fine.

#include <iostream>
#include <fstream>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>
#include <boost/serialization/split_free.hpp>
#include <boost/serialization/vector.hpp>

BOOST_SERIALIZATION_SPLIT_FREE(Mat)
namespace boost {
namespace serialization {

    /*** Mat ***/
    template<class Archive>
    void save(Archive & ar, const Mat& m, const unsigned int version)
    {
      size_t elemSize = m.elemSize(), elemType = m.type();

      ar & m.cols;
      ar & m.rows;
      ar & elemSize;
      ar & elemType; // element type.
      size_t dataSize = m.cols * m.rows * m.elemSize();

      //cout << "Writing matrix data rows, cols, elemSize, type, datasize: (" << m.rows << "," << m.cols << "," << m.elemSize() << "," << m.type() << "," << dataSize << ")" << endl;

      for (size_t dc = 0; dc < dataSize; ++dc) {
          ar & m.data[dc];
      }
    }

    template<class Archive>
    void load(Archive & ar, Mat& m, const unsigned int version)
    {
        int cols, rows;
        size_t elemSize, elemType;

        ar & cols;
        ar & rows;
        ar & elemSize;
        ar & elemType;

        m.create(rows, cols, elemType);
        size_t dataSize = m.cols * m.rows * elemSize;

        //cout << "reading matrix data rows, cols, elemSize, type, datasize: (" << m.rows << "," << m.cols << "," << m.elemSize() << "," << m.type() << "," << dataSize << ")" << endl;

        for (size_t dc = 0; dc < dataSize; ++dc) {
                  ar & m.data[dc];
        }
    }

}
}

Now, mat can be serialized and deserialized as following:

    void saveMat(Mat& m, string filename) {
            ofstream ofs(filename.c_str());
            boost::archive::binary_oarchive oa(ofs);
            //boost::archive::text_oarchive oa(ofs);
            oa << m;
    }

    void loadMat(Mat& m, string filename) {
            std::ifstream ifs(filename.c_str());
            boost::archive::binary_iarchive ia(ifs);
            //boost::archive::text_iarchive ia(ifs);
            ia >> m;
    }

I've used the binary_oarchive and binary_iarchive here to keep the memory usage down. The binary format doesn't provide portability between platforms, but if desired the text_oarchive/iarchive can be used.

Gallipot

two improvements: you could use boost::serialization::make_array instead of a custom for-loop to save individual matrix elements and loss-less zlib compression can be easily added using boost.iostreams. I've compiled a post providing full source. – Yalonda

Great post, thanks for sharing the improvements back! - I've added a link to the top of this post to make sure that people find it. – Gallipot

This won't work for non-continuous matrices, and you don't actually need to serialize elem_size because it's derived from the type of the matrix. – Psychodrama

hello I Implemented this solution but it works quite slow with zeromq library. sending 44kb image takes around 3 seconds directly connected 2 computers with 100mbit ethernet isnt it suppose to be way faster than that ? – Sondra

I'm getting error error: variable or field ‘save’ declared void using boost 1.56 and gcc 4.8.4. – Wicks

What is this strage & operator in ar & m.cols; ? – Cota

Answer 2

The earlier answers are good, but they won't work for non-continuous matrices which arise when you want to serialize regions of interest (among other things). Also, it is unnecessary to serialize elemSize() because this is derived from the type value.

Here's some code that will work regardless of continuity (with includes/namespace)

#pragma once

#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/utility.hpp>
#include <opencv2/opencv.hpp>

namespace boost {
namespace serialization {

template<class Archive>
void serialize(Archive &ar, cv::Mat& mat, const unsigned int)
{
    int cols, rows, type;
    bool continuous;

    if (Archive::is_saving::value) {
        cols = mat.cols; rows = mat.rows; type = mat.type();
        continuous = mat.isContinuous();
    }

    ar & cols & rows & type & continuous;

    if (Archive::is_loading::value)
        mat.create(rows, cols, type);

    if (continuous) {
        const unsigned int data_size = rows * cols * mat.elemSize();
        ar & boost::serialization::make_array(mat.ptr(), data_size);
    } else {
        const unsigned int row_size = cols*mat.elemSize();
        for (int i = 0; i < rows; i++) {
            ar & boost::serialization::make_array(mat.ptr(i), row_size);
        }
    }

}

} // namespace serialization
} // namespace boost

Psychodrama

This is the best, simplest and most complete answer to serialising cv::Mat. It's more complete than the solution by @TumbleCow and Christoph Heindl's blog. Thank you for sharing it! – Conscious

@Conscious No problem, glad you found it useful :) – Psychodrama

Can you provide full example of usage? – Wicks

@Wicks added to answer (also it is Archive and not Archieve :) ) – Psychodrama

Am I correct in thinking that, even though it will correctly deserialized a previously serialized non-continuous matrix, it will be deserialized as a continuous matrix, always? – Needs

Answer 3

You could use boost::serialization for that. It's heavily optimized and is pretty easy to integrate.

Possible speed-ups for your case include serializing each object as a raw binary block (see boost::serialization::make_binary) and disabling version tracking (BOOST_SERIALIZATION_DISABLE_TRACKING).

Also, you can experiment with adding compression into your serialization routines to save space (and time in case of data that is easily compressable). This can be implemented with boost::iostreams, for example.

Tribal

Answer 4

I was recently asking myself a similar question, though specifically I was trying to serialize opencv's Mat and MatND objects. Using boost::serialize is nice, but requires a couple tricks. As you don't want to go about modifying the internals of OpenCV itself to serialize these objects, you are forced to use what's called a "free" function. Since it is complicated to serialize the OpenCV objects, I found I was forced to split the serialize operation into save and load, each with a slightly different implementation. You need to use boost/serialization/split_free.hpp for this task. Boost provides good documentation for this here: http://www.boost.org/doc/libs/1_45_0/libs/serialization/doc/index.html.

Good luck!

Phoneme

To ask an impolite question: Would you be willing to share your Mat serialization code? – Gallipot

Answer 5

How about just convert your Mat to a vector and use fwrite?

The converting to vector process might affect performance, but it's safe. I suspect that all the answers above, either looping through the image data as in the accepted answer, or using make_array as in Christoph post, assume that your Mat data is contiguous, which is not necessarily the case. When your Mat data is not contiguous, the output from these answers will not be correct.

Kisung

Answer 6

I wrote this code:

/*
Will save in the file:
cols\n
rows\n
elemSize\n
type\n
DATA
*/
void serializeMatbin(Mat& mat, std::string filename){
    if (!mat.isContinuous()) {
        cout << "Not implemented yet" << endl;
        exit(1);
    }
    int elemSizeInBytes = (int)mat.elemSize();
    int elemType        = (int)mat.type();
    int dataSize        = (int)(mat.cols * mat.rows * mat.elemSize());

    FILE* FP = fopen(filename.c_str(), "wb");
    int sizeImg[4] = {mat.cols, mat.rows, elemSizeInBytes, elemType };
    fwrite(/*buffer*/ sizeImg, /*howmanyelements*/ 4, /* size of each element */ sizeof(int), /*file*/ FP);
    fwrite(mat.data, mat.cols * mat.rows, elemSizeInBytes, FP);
    fclose(FP);
}

Mat deserializeMatbin(std::string filename){
    FILE* fp = fopen(filename.c_str(), "r");
    int header[4];
    fread(header, sizeof(int), 4, fp);
    int cols = header[0]; 
    int rows = header[1];
    int elemSizeInBytes = header[2];
    int elemType = header[3];

    Mat outputMat = Mat(rows, cols, elemType);

    fread(outputMat.data, elemSizeInBytes, cols * rows, fp);
    fclose(fp);
    return outputMat;
}

void testSerializeMatbin(){
    Mat a = Mat::ones(/*cols*/ 10, /* rows */ 5, CV_8U) * 2;
    std::string filename = "test.matbin";
    serializeMatbin(a, filename);
    Mat b = deserializeMatbin(filename);
    cout << "Rows: " << b.rows << " Cols: " << b.cols << " type: " << b.type()<< endl;
}

Cota

Answer 7

You can use msgpack also Create an adaptor https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor Here is example code. It might be useful:

namespace clmdep_msgpack {
MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS) {
    namespace adaptor {

        //I am sending as bin (int)(int)(int)(char*)
        //Mat values:         rows,cols,type,data
        template<>
        struct convert<cv::Mat> {
            clmdep_msgpack::object const &operator()(clmdep_msgpack::object const &o, cv::Mat &v) const
            {
                if(o.type != clmdep_msgpack::type::BIN) throw clmdep_msgpack::type_error();
                char *buffer = (char *) o.via.bin.ptr;
                int buffer_size = o.via.bin.size;
                int rows, cols, type;
                rows = *reinterpret_cast<int *>(buffer);
                cols = *reinterpret_cast<int *>(buffer + 1 * sizeof(int));
                type = *reinterpret_cast<int *>(buffer + 2 * sizeof(int));
                cv::Mat(rows, cols, type, (void *) (buffer + 3 * sizeof(int))).copyTo(v);

                return o;
            }
        };

        template<>
        struct pack<cv::Mat> {
            template<typename Stream>
            clmdep_msgpack::packer<Stream> &operator()(clmdep_msgpack::packer<Stream> &o, cv::Mat const &v) const
            {
                // packing member variables as bin.
                size_t mat_size;
                if(v.isContinuous()) {
                    mat_size = v.total() * v.elemSize();
                } else {
                    mat_size = v.step[v.dims - 1];
                    for(int t = 0; t < v.dims; t++) {
                        // calculate total size of multi dimensional matrix by multiplying dimensions
                        mat_size *= v.size[t];
                    }
                }
                int extra_ints = 3;
                int buffer_size = extra_ints * sizeof(int) + mat_size;
                // Allocate destination image buffer
                char *imagebuffer = new char[buffer_size];
                int type = v.type();
                std::memcpy(imagebuffer, &(v.rows), sizeof(int));
                std::memcpy(imagebuffer + 1 * sizeof(int), &(v.cols), sizeof(int));
                std::memcpy(imagebuffer + 2 * sizeof(int), &type, sizeof(int));

                if(v.isContinuous()) {
                    std::memcpy((imagebuffer + 3 * sizeof(int)), (char *) v.data, mat_size);
                } else {
                    const size_t rowsize = v.step[v.dims - 1] * v.size[v.dims - 1];
                    size_t coordinates[v.dims - 1] = {0};
                    size_t srcptr = 0, dptr = extra_ints * sizeof(int);
                    while(dptr < buffer_size) {
                        // we copy entire rows at once, so lowest iterator is always [dims-2]
                        // this is legal since OpenCV does not use 1 dimensional matrices internally (a 1D matrix is a 2d matrix with only 1 row)
                        std::memcpy(&imagebuffer[dptr], &(((char *) v.data)[srcptr]), rowsize);
                        // destination matrix has no gaps so rows follow each other directly
                        dptr += rowsize;
                        // src matrix can have gaps so we need to calculate the address of the start of the next row the hard way
                        // see *brief* text in opencv2/core/mat.hpp for address calculation
                        coordinates[v.dims - 2]++;
                        srcptr = 0;
                        for(int t = v.dims - 2; t >= 0; t--) {
                            if(coordinates[t] >= v.size[t]) {
                                if(t == 0) break;
                                coordinates[t] = 0;
                                coordinates[t - 1]++;
                            }
                            srcptr += v.step[t] * coordinates[t];
                        }
                    }
                }
                o.pack_bin(buffer_size);
                o.pack_bin_body(imagebuffer, buffer_size);
                return o;
            }
        };

        template<>
        struct object_with_zone<cv::Mat> {
            void operator()(clmdep_msgpack::object::with_zone &o, cv::Mat const &v) const
            {
                size_t mat_size;
                if(v.isContinuous()) {
                    mat_size = v.total() * v.elemSize();
                } else {
                    mat_size = v.step[v.dims - 1];
                    for(int t = 0; t < v.dims; t++) {
                        // calculate total size of multi dimensional matrix by multiplying dimensions
                        mat_size *= v.size[t];
                    }
                }
                int extra_ints = 3;
                int buffer_size = extra_ints * sizeof(int) + mat_size;
                // Allocate destination image buffer
                char *imagebuffer = new char[buffer_size];
                int type = v.type();
                std::memcpy(imagebuffer, &(v.rows), sizeof(int));
                std::memcpy(imagebuffer + 1 * sizeof(int), &(v.cols), sizeof(int));
                std::memcpy(imagebuffer + 2 * sizeof(int), &type, sizeof(int));

                if(v.isContinuous()) {
                    std::memcpy((imagebuffer + 3 * sizeof(int)), (char *) v.data, mat_size);
                } else {
                    const size_t rowsize = v.step[v.dims - 1] * v.size[v.dims - 1];
                    size_t coordinates[v.dims - 1] = {0};
                    size_t srcptr = 0, dptr = extra_ints * sizeof(int);
                    while(dptr < buffer_size) {
                        // we copy entire rows at once, so lowest iterator is always [dims-2]
                        // this is legal since OpenCV does not use 1 dimensional matrices internally (a 1D matrix is a 2d matrix with only 1 row)
                        std::memcpy(&imagebuffer[dptr], &(((char *) v.data)[srcptr]), rowsize);
                        // destination matrix has no gaps so rows follow each other directly
                        dptr += rowsize;
                        // src matrix can have gaps so we need to calculate the address of the start of the next row the hard way
                        // see *brief* text in opencv2/core/mat.hpp for address calculation
                        coordinates[v.dims - 2]++;
                        srcptr = 0;
                        for(int t = v.dims - 2; t >= 0; t--) {
                            if(coordinates[t] >= v.size[t]) {
                                if(t == 0) break;
                                coordinates[t] = 0;
                                coordinates[t - 1]++;
                            }
                            srcptr += v.step[t] * coordinates[t];
                        }
                    }
                }
                o.type = type::BIN;
                o.via.bin.size = buffer_size;
                o.via.bin.ptr = imagebuffer;
            }
        };


    } // namespace adaptor
} // MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS)
} // names

Elater