Eigen and boost::serialize
Asked Answered
A

3

11

I tried to write a generic serialize function which takes any dense matrix and serializes it: Some other questions which help but not to the end are here: Question1 Question2

I tried the following which should work:

namespace boost {
namespace serialization {
    template<class Archive, typename Derived> void serialize(Archive & ar,  Eigen::EigenBase<Derived> & g, const unsigned int version)
    {
        ar & boost::serialization::make_array(g.derived().data(), g.size());
    }
    }; // namespace serialization
}; // namespace boost

When I try to serialize an Eigen::Matrix<double,4,4>

Eigen::Matrix<double,4,4> a; 
boost::serialize(ar, a);

The compiler can somehow not match the template above? And the following errors are given :

/usr/local/include/boost/serialization/access.hpp|118|error: 'class Eigen::Matrix' has no member named 'serialize'|

Applegate answered 22/8, 2013 at 13:53 Comment(0)
H
7

I make use of Eigen's plugin based extension:

/**
 * @file EigenDenseBaseAddons.h
 */
#ifndef EIGEN_DENSE_BASE_ADDONS_H_
#define EIGEN_DENSE_BASE_ADDONS_H_

friend class boost::serialization::access;
template<class Archive>
void save(Archive & ar, const unsigned int version) const {
  derived().eval();
  const Index rows = derived().rows(), cols = derived().cols();
  ar & rows;
  ar & cols;
  for (Index j = 0; j < cols; ++j )
    for (Index i = 0; i < rows; ++i )
      ar & derived().coeff(i, j);
}

template<class Archive>
void load(Archive & ar, const unsigned int version) {
  Index rows, cols;
  ar & rows;
  ar & cols;
  if (rows != derived().rows() || cols != derived().cols() )
    derived().resize(rows, cols);
  ar & boost::serialization::make_array(derived().data(), derived().size());
}

template<class Archive>
void serialize(Archive & ar, const unsigned int file_version) {
  boost::serialization::split_member(ar, *this, file_version);
}

#endif // EIGEN_DENSE_BASE_ADDONS_H_

Configure Eigen to use this pulgin:(simply define the macro before including any Eigen header)

#ifndef EIGEN_CONFIG_H_
#define EIGEN_CONFIG_H_

#include <boost/serialization/array.hpp>
#define EIGEN_DENSEBASE_PLUGIN "EigenDenseBaseAddons.h"

#include <Eigen/Core>

#endif // EIGEN_CONFIG_H_

Though I have not really tested this throughly, it works well and can also deal with Array or any other dense Eigen objects. It also works perfectly for expressions like vec.tail<4>() but may fail (without any compile error) for expression like mat.topRows<2>() or block operations. (See update: now works for sub matrices also)

In comparison to the other current answer, this works for more set of expression and might avoid some temporary. A non-intrusive version is also probably possible by passing PlainObjectBase<Derived> objects to the serialize functions..


/// Boost Serialization Helper

template <typename T>
bool serialize(const T& data, const std::string& filename) {
  std::ofstream ofs(filename.c_str(), std::ios::out);
  if (!ofs.is_open())
    return false;
  {
    boost::archive::binary_oarchive oa(ofs);
    oa << data;
  }
  ofs.close();
  return true;
}

template <typename T>
bool deSerialize(T& data, const std::string& filename) {
  std::ifstream ifs(filename.c_str(), std::ios::in);
  if (!ifs.is_open())
    return false;
  {
    boost::archive::binary_iarchive ia(ifs);
    ia >> data;
  }
  ifs.close();
  return true;
}

And some test code:

VectorXf vec(100);
vec.setRandom();
serializeText(vec.tail<5>(), "vec.txt");

MatrixXf vec_in;
deSerialize(vec_in, "vec.bin");
assert(vec_in.isApprox(vec.tail<5>()));

serialize(Vector2f(0.5f,0.5f), "a.bin");
Vector2f a2f;
deSerializeBinary(a2f, "a.bin");
assert(a2f.isApprox(Vector2f(0.5f,0.5f)));
VectorXf axf;
deSerialize(axf, "a.bin");
assert(aXf.isApprox(Vector2f(0.5f,0.5f)));

boost::shared_ptr<Vector4f> b = boost::make_shared<Vector4f>(Vector4f::Random());
serialize(b, "b.tmp");
boost::shared_ptr<Vector4f> b_in;
deSerialize(b_in, "b.tmp");
BOOST_CHECK_EQUAL(*b, *b_in);

Matrix4f m(Matrix4f::Random());
serialize(m.topRows<2>(), "m.bin");
deSerialize(m_in, "m.bin");

Update: I made some minor modifications,now serialization of sub matrices also works.

Herzl answered 1/5, 2014 at 11:59 Comment(0)
G
11

I've tested out your code and it also did not work when I tried to compile it. However, based on the documentation for Boost Serialize, I am under the impression that it is intended to be used with the stream operator <<. The following code works fine for me:

namespace boost {
   namespace serialization {
      template <class Archive, typename Derived> 
      void serialize( Archive & ar, Eigen::EigenBase<Derived> & g, const unsigned int version){
          ar & boost::serialization::make_array(g.derived().data(), g.size());
      }
   }
}

int main (int argc, char* argv[]){
    std::ofstream out("my_archive");
    boost::archive::text_oarchive oa (out);

    Eigen::Matrix <double, 4, 4> a;
    out << a;
    return 0;
}

The file my_archive is created in the working folder with non-zero values (just the uninitialized garbage in memory, with the above over-simplified code).

EDIT: I tried using the exact code above in my own application and found that I received the same error as you did. I honestly do not understand why that it, right now. The simplest fix that I found was to replace Eigen::EigenBase<Derived> with the actual Matrix type being used. The current code that I am using is:

namespace boost{
    namespace serialization {
        template <class Archive, typename Scalar>
        void serialize ( Archive & ar, Eigen::Matrix<Scalar, -1, -1, 0, -1, -1> & g, const unsigned int version ){ /* ... */ }
     }
}

The above code works for any scalar type (float, double, int) and for dynamic/runtime sized matrixes. For static sized, check into and update the template parameters accordingly.

EDIT #2 (April 09, 2014):

Despite the appearance of the above code working, when I tried to integrate it fully into my code and exercise it with appropriate unit testing, it stopped working. Unfortunately, the error messages that I was being given -- both from Visual Studio and clang -- were most unhelpful. Fortunately, gcc had buried within the horrible mess of error messages, a reference to CV-mismatch, which seems to have allowed me to fully address this.

The following code now appears to compile and run successfully. I've tried to make the formatting legible (without side-scrolling) -- hopefully the code below is clear:

namespace boost{
    namespace serialization{

        template<   class Archive, 
                    class S, 
                    int Rows_, 
                    int Cols_, 
                    int Ops_, 
                    int MaxRows_, 
                    int MaxCols_>
        inline void save(
            Archive & ar, 
            const Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & g, 
            const unsigned int version)
            {
                int rows = g.rows();
                int cols = g.cols();

                ar & rows;
                ar & cols;
                ar & boost::serialization::make_array(g.data(), rows * cols);
            }

        template<   class Archive, 
                    class S, 
                    int Rows_,
                    int Cols_,
                    int Ops_, 
                    int MaxRows_, 
                    int MaxCols_>
        inline void load(
            Archive & ar, 
            Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & g, 
            const unsigned int version)
        {
            int rows, cols;
            ar & rows;
            ar & cols;
            g.resize(rows, cols);
            ar & boost::serialization::make_array(g.data(), rows * cols);
        }

        template<   class Archive, 
                    class S, 
                    int Rows_, 
                    int Cols_, 
                    int Ops_, 
                    int MaxRows_, 
                    int MaxCols_>
        inline void serialize(
            Archive & ar, 
            Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & g, 
            const unsigned int version)
        {
            split_free(ar, g, version);
        }


    } // namespace serialization
} // namespace boost

A few critical points on the above code:

  • This code now has templated parameters for all of Eigen's Matrix parameters. This should allow it to work with all types of matrices and vectors, whether sized at compile-time or run-time. This is a major enhancement over the above code

  • It is essential that the serialization code be split into separate save and load functions. Otherwise, the deserialization code will not resize the matrix to hold the original data. I believe that Boost::Serialize does provide some additional functions that can be overloaded to perform operations on serialization or deserialization, but this approach was easier to implement.

  • The const qualifier on the save method is essential. This was the source of my troubles before an obscure g++ error keyed me in on this.

  • I cannot say that I have fully stress-tested this code yet. If you (or anyone else) finds any more problems with it, please do let me know and I'll try and follow up with anything else that I find.

Trust that that helps.

Shmuel

Guideboard answered 7/4, 2014 at 3:20 Comment(6)
Good Comment :-)!: I still have no answer, why the compliler can not match it... :-)Applegate
@Gabriel, I'm not sure exactly what error messages your compiler is providing. From my own (extremely frustrating) attempts at getting this to work, I'll note that the error messages which I received were extremely unhelpful. Please take another look at my answer, as I have updated it again with the results of my own attempts to make it work.Guideboard
@Gabriel, can you please let me know whether the updated version of code in my answer has solved your problem? I am presently using the exact code above in my own application (with diff. namespaces) and it now works flawlessly. If it is working for you, I'd appreciate if you could accept this answer. Thanks!Guideboard
Thanks, yeah this code is good so far: The major disadvantages for me is that it cannot handle Eigen::Base types and other only Eigen:.Matrix types, would be cool to allow other types instead of only Matrix class, lets say we want to serialize m.tail<4>(). But this will not work I think because of ADL i think...? I will test your generalized code in my framework soon!Applegate
If you are going to serialize using the backend data() buffer, don't you also need to worry about the stride and the storage order (aka flags)? If not, then you may serialize a row ordered matrix and deserialize a column ordered one?Expendable
@Phil, in all honesty, I hadn't looked into this during the time I was actively using Eigen on my own project, as my code was using column-major matrices exclusively at the time and had no particular need to investigate these flags. I am no longer using Eigen at this time, for other reasons. With that in mind, I am not particularly familiar with the nuances, flags, etc., although from what I can recall from browsing the documentation, this solution does not address those issues. A fully general solution would need to accommodate and guard against these issues.Guideboard
H
7

I make use of Eigen's plugin based extension:

/**
 * @file EigenDenseBaseAddons.h
 */
#ifndef EIGEN_DENSE_BASE_ADDONS_H_
#define EIGEN_DENSE_BASE_ADDONS_H_

friend class boost::serialization::access;
template<class Archive>
void save(Archive & ar, const unsigned int version) const {
  derived().eval();
  const Index rows = derived().rows(), cols = derived().cols();
  ar & rows;
  ar & cols;
  for (Index j = 0; j < cols; ++j )
    for (Index i = 0; i < rows; ++i )
      ar & derived().coeff(i, j);
}

template<class Archive>
void load(Archive & ar, const unsigned int version) {
  Index rows, cols;
  ar & rows;
  ar & cols;
  if (rows != derived().rows() || cols != derived().cols() )
    derived().resize(rows, cols);
  ar & boost::serialization::make_array(derived().data(), derived().size());
}

template<class Archive>
void serialize(Archive & ar, const unsigned int file_version) {
  boost::serialization::split_member(ar, *this, file_version);
}

#endif // EIGEN_DENSE_BASE_ADDONS_H_

Configure Eigen to use this pulgin:(simply define the macro before including any Eigen header)

#ifndef EIGEN_CONFIG_H_
#define EIGEN_CONFIG_H_

#include <boost/serialization/array.hpp>
#define EIGEN_DENSEBASE_PLUGIN "EigenDenseBaseAddons.h"

#include <Eigen/Core>

#endif // EIGEN_CONFIG_H_

Though I have not really tested this throughly, it works well and can also deal with Array or any other dense Eigen objects. It also works perfectly for expressions like vec.tail<4>() but may fail (without any compile error) for expression like mat.topRows<2>() or block operations. (See update: now works for sub matrices also)

In comparison to the other current answer, this works for more set of expression and might avoid some temporary. A non-intrusive version is also probably possible by passing PlainObjectBase<Derived> objects to the serialize functions..


/// Boost Serialization Helper

template <typename T>
bool serialize(const T& data, const std::string& filename) {
  std::ofstream ofs(filename.c_str(), std::ios::out);
  if (!ofs.is_open())
    return false;
  {
    boost::archive::binary_oarchive oa(ofs);
    oa << data;
  }
  ofs.close();
  return true;
}

template <typename T>
bool deSerialize(T& data, const std::string& filename) {
  std::ifstream ifs(filename.c_str(), std::ios::in);
  if (!ifs.is_open())
    return false;
  {
    boost::archive::binary_iarchive ia(ifs);
    ia >> data;
  }
  ifs.close();
  return true;
}

And some test code:

VectorXf vec(100);
vec.setRandom();
serializeText(vec.tail<5>(), "vec.txt");

MatrixXf vec_in;
deSerialize(vec_in, "vec.bin");
assert(vec_in.isApprox(vec.tail<5>()));

serialize(Vector2f(0.5f,0.5f), "a.bin");
Vector2f a2f;
deSerializeBinary(a2f, "a.bin");
assert(a2f.isApprox(Vector2f(0.5f,0.5f)));
VectorXf axf;
deSerialize(axf, "a.bin");
assert(aXf.isApprox(Vector2f(0.5f,0.5f)));

boost::shared_ptr<Vector4f> b = boost::make_shared<Vector4f>(Vector4f::Random());
serialize(b, "b.tmp");
boost::shared_ptr<Vector4f> b_in;
deSerialize(b_in, "b.tmp");
BOOST_CHECK_EQUAL(*b, *b_in);

Matrix4f m(Matrix4f::Random());
serialize(m.topRows<2>(), "m.bin");
deSerialize(m_in, "m.bin");

Update: I made some minor modifications,now serialization of sub matrices also works.

Herzl answered 1/5, 2014 at 11:59 Comment(0)
P
7

Here is a more general and shorter version with the following features:

  • works correctly whether the matrix is row or column-major
  • uses name-value-pairs so it works with xml archives and the like
  • does not need the split between save and load version
  • adds a wrapper function for Eigen::Transform (Affine3d, Isometry3f etc.)
  • unit-tested for various combinations, e.g. it works when you save a column-major 2x2 MatrixXd and load it as a row-major Eigen::Matrix when using XML archives. However this does apparently not work for binary or text archives! In these cases the types must match exactly

Code:

namespace boost { namespace serialization {

template<   class Archive,
            class S,
            int Rows_,
            int Cols_,
            int Ops_,
            int MaxRows_,
            int MaxCols_>
inline void serialize(Archive & ar,
                      Eigen::Matrix<S, Rows_, Cols_, Ops_, MaxRows_, MaxCols_> & matrix,
                      const unsigned int version)
{
  int rows = matrix.rows();
  int cols = matrix.cols();
  ar & make_nvp("rows", rows);
  ar & make_nvp("cols", cols);    
  matrix.resize(rows, cols); // no-op if size does not change!

  // always save/load row-major
  for(int r = 0; r < rows; ++r)
    for(int c = 0; c < cols; ++c)
      ar & make_nvp("val", matrix(r,c));
}    

template<   class Archive,
            class S,
            int Dim_,
            int Mode_,
            int Options_>
inline void serialize(Archive & ar,
                      Eigen::Transform<S, Dim_, Mode_, Options_> & transform,
                      const unsigned int version)
{
  serialize(ar, transform.matrix(), version);
}    
}} // namespace boost::serialization
Pannikin answered 29/1, 2016 at 0:22 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.