Converting data from glReadPixels() to OpenCV::Mat

Question

I want to get every OpenGL frame from an animation with glReadPixels() and convert the data to OpenCV::Mat. I know that glReadPixels() gets the data by rows from the lower one to upper one, from left to right. On the other hand, OpenCV stores the data differently.

Does anybody know any library or any tutorial/example that helps me to convert data from glReadPixels to a OpenCV:Mat in C++?

SUMMARY

 OpenGL frame      ----------------------->        CV::Mat

Data from left to right,                    Data from left to right,
bottom to top.                              top to bottom.

Daubigny

Answer 1

First we create an empty (or unititialized) cv::Mat for our data to be read into directly. This can be done once at startup, but on the other hand cv::Mat::create doesn't really cost much when the image already has matching size and type. The type depends on your needs, usually it's something like CV_8UC3 for a 24-bit color image.

cv::Mat img(height, width, CV_8UC3);

or

img.create(height, width, CV_8UC3);

Then you have to account for cv::Mat not neccessarily storing image rows contiguously. There might be a small padding value at the end of each row to make rows 4-byte aligned (or 8?). So you need to mess with the pixel storage modes:

//use fast 4-byte alignment (default anyway) if possible
glPixelStorei(GL_PACK_ALIGNMENT, (img.step & 3) ? 1 : 4);

//set length of one complete row in destination data (doesn't need to equal img.cols)
glPixelStorei(GL_PACK_ROW_LENGTH, img.step/img.elemSize());

Next, the type of the matrix influences the format and type parameters of glReadPixels. If you want color images you have to keep in mind that OpenCV usually stores color values in BGR order, so you need to use GL_BGR(A) (which were added with OpenGL 1.2) instead of GL_RGB(A). For one component images use either GL_LUMINANCE (which sums the individual color components) or GL_RED, GL_GREEN, ... (to get an individual component). So for our CV_8UC3 image the final call to read it directly into the cv::Mat would be:

glReadPixels(0, 0, img.cols, img.rows, GL_BGR, GL_UNSIGNED_BYTE, img.data);

Finally, OpenCV stores images from top to bottom. So you may need to either flip them after getting them or render them flipped in OpenGL in the first place (this can be done by adjusting the projection matrix, but keep an eye on triangle orientation in this case). To flip a cv::Mat vertically, you can use cv::flip:

cv::flip(img, flipped, 0);

So to keep in mind OpenCV:

• stores images from top to bottom, left to right
• stores color images in BGR order
• might not store image rows tightly packed

Emilio

Really really nice answer. I was already working on the flipping but I didn't pay attention to glPixelStorei. Thanks – Daubigny

I cannot call GL_BGR, I don't know why the linker cannot find it because I have got the latest version – Daubigny

@Daubigny It was introduced with OpenGL 1.2. So if your hardware and driver support it (which is very likely, unless you bought your graphics card 20 years ago), you just need to define this symbol, by including a reasonably new glext.h or by using an extension managment library in the first place (like GLEW), which should bring its own header with constant definitions. This would be necessary once you want to use any functionality higher than 1.1, anyway (like PBOs, which might speed up you reading performance whe used properly). – Emilio

Yeah, definetely glext.h fixes the problem. Now I got everything working with right colors. Thanks a lot. – Daubigny

This comment is still being useful, if I could I would upvote it 20 times. – Daubigny

I tried this today and it returns a gray image. Any ideias? btw, GL_BGR is not defined, only GL_BGR_EXT (under windows, MSVC 2013) – Everard

@Everard It should be defined if you have support for GL 1.2. And did you make sure the image format matches? – Emilio

@ChristianRau I had to call wglMakeCurrent() as I am on Windows, MFC. Plus, I had to use GL_BGR_EXT instead of GL_BGR – Everard

If anyone wonders why this doesn't work is because cv::Mat::create only creates a header, you need to clone an existing Mat or initialize it with some values with cv::Mat::zeros for example. – Repeated

@GáborFekete No, create does create storage for the matrix as well. It might not initialize that storage, but you don't care what's in it anyway, since you're you completely overwrite it in the next step anyway. But it's definitely allocated. – Emilio

Answer 2

unsigned char* getPixelData( int x1, int y1, int x2, int y2 )
{
    int y_low, y_hi;
    int x_low, x_hi;

    if ( y1 < y2 )
    {
        y_low = y1;
        y_hi  = y2;
    }
    else
    {
        y_low = y2;
        y_hi  = y1;
    }

    if ( x1 < x2 )
    {
        x_low = x1;
        x_hi  = x2;
    }
    else
    {
        x_low = x2;
        x_hi  = x1;
    }

    while ( glGetError() != GL_NO_ERROR )
    {
        ;
    }

    glReadBuffer( GL_BACK_LEFT );

    glDisable( GL_TEXTURE_2D );

    glPixelStorei( GL_PACK_ALIGNMENT, 1 );

    unsigned char *data = new unsigned char[ ( x_hi - x_low + 1 ) * ( y_hi - y_low + 1 ) * 3 ];

    glReadPixels( x_low, y_low, x_hi-x_low+1, y_hi-y_low+1, GL_RGB, GL_UNSIGNED_BYTE, data );

    if ( glGetError() != GL_NO_ERROR )
    {
        delete[] data;
        return 0;
    }
    else
    {
        return data;
    }
}

use:

CvSize size = cvSize( 320, 240 );

unsigned char *pixel_buf = getPixelData( 0, 0, size.width - 1, size.height - 1 );

if ( pixel_buf == 0 )
    return 0;

IplImage *result = cvCreateImage( size, IPL_DEPTH_8U, 3 );
memcpy( result->imageData, pixel_buf, size.width * size.height * 3 );
delete[] pixel_buf;

Bunion