OpenCV CV::Mat and Eigen::Matrix

Question

Is there a reversible way to convert an OpenCV cv::Mat object to an Eigen::Matrix?

e.g., Some way of doing:

cv::Mat cvMat;
Eigen::Matrix eigMat;
camera->retrieve(cvMat);

// magic to convert cvMat to eigMat
// work on eigMat
// convert eigMat back to cvMat

imshow("Image", cvMat);

I've tried using cv2eigen and eigen2cv, but the resulting cvMat is completely mangled and I'm not exactly sure why. The dimensions are correct, but the graphics are totally trashed, so possibly a bytes-per-pixel or datasize issue?

Tupper

Answer 1

You should consider using Eigen::Map to wrap OpenCV matrices in order to be used directly by the Eigen SDK. This allows you to apply almost all functionalities implemented in Eigen on matrix allocated by OpenCV

In particular you simply instantiate an Eigen::Map providing the pointer to the cv::Mat buffer:

//allocate memory for a 4x4 float matrix
cv::Mat cvT(4,4,CV_32FC1); 

//directly use the buffer allocated by OpenCV
Eigen::Map<Matrix4f> eigenT( cvT.data() ); 

for more information on Eigen::Map take a look at Eigen Tutorial: Map Class

Aceous

Perfect, that's pretty much exactly what I was looking for. When using a multi-channel image (RGB, YUV, or any other combination of channels), how would you best convert it? A separate matrix per channel? Into a 3D matrix of widthXheightXchannels? Or just expand it by width (width*3)*height ? – Tupper

Mulit-channel images are typically stored as interleaved arrays (e.g RGBRGBRGB...). Depending on what you want to do with them you might consider mapping each single channel to a different Eigen::Map exploiting the stride parameter: cv::Mat cvT(4,4,CV_32FC3); //a 3 channel float matrix Eigen::Map<Matrix4f, RowMajor, Stride<3,1>> red( cvT.data ); Eigen::Map<Matrix4f, RowMajor, Stride<3,1>> green( cvT.data +1 ); Eigen::Map<Matrix4f, RowMajor, Stride<3,1>> blue( cvT.data +2 ); – Aceous

Using cvT.data() does not work for me, gives a compiler error. I posted an answer on how I do it below, including information how to do it for arbitrary sized matrices and with the reverse conversion from Eigen to OpenCV. – Eraste

Hi! @Aceous is right Eigen::Map is a good starting point. I've created a simple eigen2cv wrapper that allows mapping Eigen types to OpenCV and vice versa. It supports read/write mapping to/from OpenCV and it also supports read-only mapping from Eigen expressions. A more detailed overview and implementation can be found in my post: computer-vision-talks.com/articles/mapping-eigen-to-opencv. – Feldspar

@Eraste This code worked for me for OpenCV 3.4.1 and Eigen 3.3.4 for a CV_32FC1 cv::Mat: Eigen::Map<Eigen::Matrix4f> eigen_mat(reinterpret_cast<float*>(cv_mat.data)); – Sake

@Sake Would be much better to use cv_mat.ptr<float>() though. And be careful, I think you're mapping row-major data to a column-major matrix - I think your result might be wrong, it'll be the transpose. I think you need Eigen::RowMajor. – Eraste

@Ela782. Thanks I actually had tansposed matrices because of that. I am now using this (also using opencv's MatX): cv::Matx33f cv_mat; Eigen::Map<Eigen::Matrix<float,3,3,Eigen::RowMajor>> eigen_mat(cv_mat.val); – Sake

Answer 2

You can also use

void eigen2cv(const Eigen::Matrix<_Tp, _rows, _cols, _options, _maxRows, _maxCols>& src, Mat& dst)

and

void cv2eigen(const Mat& src, Eigen::Matrix<_Tp, _rows, _cols, _options, _maxRows, _maxCols>& dst)

from #include <opencv2/core/eigen.hpp>.

Preconscious

Answer 3

You can map arbitrary matrices between Eigen and OpenCV (without copying data).

You have to be aware of two things though:

• Eigen defaults to column-major storage, OpenCV stores row-major. Therefore, use the Eigen::RowMajor flag when mapping OpenCV data.

• The OpenCV matrix has to be continuous (i.e. ocvMatrix.isContinuous() needs to be true). This is the case if you allocate the storage for the matrix in one go at the creation of the matrix (e.g. as in my example below, or if the matrix is the result of a operation like Mat W = A.inv();)

Example:

Mat A(20, 20, CV_32FC1);
cv::randn(A, 0.0f, 1.0f); // random data

// Map the OpenCV matrix with Eigen:
Eigen::Map<Eigen::Matrix<float, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor>> A_Eigen(A.ptr<float>(), A.rows, A.cols);

// Do something with it in Eigen, create e.g. a new Eigen matrix:
Eigen::Matrix<float, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor> B = A_Eigen.inverse();

// create an OpenCV Mat header for the Eigen data:
Mat B_OpenCV(B.rows(), B.cols(), CV_32FC1, B.data());

For multi-channel matrices (e.g. images), you can use 'Stride' exactly as Pierluigi suggested in his comment!

Eraste

This is very useful. The other option require compiling opencv with the flag WITH_EIGEN which maybe not be possible to enforce organisation wide. – Asgard

Answer 4

This works for me,

  #include <opencv2/core/eigen.hpp>

  cv::Mat image;
  image = cv::imread("/dataset/images/15207_angle_image.jpg", CV_LOA D_IMAGE_GRAYSCALE);   // Read the file
  Eigen::Matrix<float,Eigen::Dynamic, Eigen::Dynamic> eigen_mat;
  cv::cv2eigen(image, eigen_mat);

Taligrade

Answer 5

Pierluigi's version has not worked for me completely for 3 channel images! After some investigation I ended with the following solution which has worked for me:

using namespace Eigen;

constexpr uint32_t height = 3;
constexpr uint32_t width = 7;

cv::Mat img(height, width, CV_32FC3, cv::Scalar(1.0f, 2.0f, 3.0f));

using MatrixXfRowMajor = Matrix<float, Dynamic, Dynamic, RowMajor>;
using C3Stride = Stride<Dynamic, 3>;
C3Stride c3Stride(width *3,3);


using cvMap = Map<MatrixXfRowMajor, Unaligned, C3Stride >;
cvMap imgC1(reinterpret_cast<float*>(img.data) + 0, img.rows, img.cols, c3Stride);
cvMap imgC2(reinterpret_cast<float*>(img.data) + 1, img.rows, img.cols, c3Stride);
cvMap imgC3(reinterpret_cast<float*>(img.data) + 2, img.rows, img.cols, c3Stride);

std::cout << imgC1 << std::endl << std::endl;
std::cout << imgC2 << std::endl << std::endl;
std::cout << imgC3 << std::endl << std::endl;

Admire