convert QVideoFrame to QImage

Question

I want to get every frames from a QMediaPlayer and convert it to QImage (or cv::Mat)

so I used videoFrameProbed signal from QVideoProbe:

connect(&video_probe_, &QVideoProbe::videoFrameProbed, 
         [this](const QVideoFrame& currentFrame){
   //QImage img = ??
}

But I didn't find any way for getting QImage from QVideoFrame!

How can I convert QVideoFrame to QImage ?!

Coady

That depends on the format of the QVideoFrame. QImage can only accept a couple of RGB-based formats. There are various libraries that can convert between various YUV and RGB formats. But if you can I'd recommend reading your video frames with OpenCV directly. – Leeannaleeanne

Did you resolve this? – Bouie

Answer 1

You can use QImage's constructor:

 QImage img( currentFrame.bits(),
             currentFrame.width(),
             currentFrame.height(),
             currentFrame.bytesPerLine(),
             imageFormat);

Where you can get imageFormat from pixelFormat of the QVideoFrame:

 QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(currentFrame.pixelFormat());

Fraudulent

though the video is likely in YUV format (imageFormatFromPixelFormat will fail). Not sure if Qt has yuv conversion built in.. alternatively could use libyuv. – Scholium

I think Qt relies on underlying mediaframework for conversions. On Linux that is gstreamer which supports a lot of conversions. As a supporting fact, in custom implementations of QAbstractVideoSurface you are forced to implement pure virtual supportedPixelFormats, and when I specified BGR24 as the only format that is the format I received. – Interposition

Please also read requirements here: doc.qt.io/qt-5/qvideoframe.html#details for QVideoFrame to call map() before calling bits() – Interposition

I tried, then img.save("filename.png"), but there are not output. – Bouie

Answer 2

For QCamera output, that method doesn't always work. In particular, QVideoFrame::imageFormatFromPixelFormat() returns QImage::Format_Invalid when given QVideoFrame::Format_Jpeg, which is what's coming out of my QCamera. But this works:

QImage Camera::imageFromVideoFrame(const QVideoFrame& buffer) const
{
    QImage img;
    QVideoFrame frame(buffer);  // make a copy we can call map (non-const) on
    frame.map(QAbstractVideoBuffer::ReadOnly);
    QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(
                frame.pixelFormat());
    // BUT the frame.pixelFormat() is QVideoFrame::Format_Jpeg, and this is
    // mapped to QImage::Format_Invalid by
    // QVideoFrame::imageFormatFromPixelFormat
    if (imageFormat != QImage::Format_Invalid) {
        img = QImage(frame.bits(),
                     frame.width(),
                     frame.height(),
                     // frame.bytesPerLine(),
                     imageFormat);
    } else {
        // e.g. JPEG
        int nbytes = frame.mappedBytes();
        img = QImage::fromData(frame.bits(), nbytes);
    }
    frame.unmap();
    return img;
}

Subcartilaginous

Answer 3

Qt has a private function qt_imageFromVideoFrame for converting QVideoFrame to QImage. If you want to use it, you'll need to:

1. Add QT += multimedia multimedia-private to your .pro file
2. Add #include "private/qvideoframe_p.h" to your .cpp file

3. Then you can use qt_imageFromVideoFrame with the following signature:

   QImage qt_imageFromVideoFrame( const QVideoFrame& f );

Be aware of the warning in the header:

//
//  W A R N I N G
//  -------------
//
// This file is not part of the Qt API.  It exists purely as an
// implementation detail.  This header file may change from version to
// version without notice, or even be removed.
//
// We mean it.
//

The implementation uses QVideoFrame::map() and does low-level manipulation with the bits. The converters handle a variety of conditions including YUV.

The qt_imageFromVideoFrame works on almost all platforms. i.e. Windows, macOS, Linux and iOS. The only platform it doesn't work reliably well is Android. There, you'll need to use OpenGL calls to extract the VideoFrame. Also, on Android, you need to call QImage:rgbSwapped() at the end because QImage stores images as ARGB whereas OpenGL retrieves them as ABGR.

QImage QVideoFrameToQImage( const QVideoFrame& videoFrame )
{
    if ( videoFrame.handleType() == QAbstractVideoBuffer::NoHandle )
    {
        QImage image = qt_imageFromVideoFrame( videoFrame );
        if ( image.isNull() ) return QImage();
        if ( image.format() != QImage::Format_ARGB32 ) return image.convertToFormat( QImage::Format_ARGB32 );
        return image;
    }
    if ( videoFrame.handleType() == QAbstractVideoBuffer::GLTextureHandle )
    {
        QImage image( videoFrame.width(), videoFrame.height(), QImage::Format_ARGB32 );
        GLuint textureId = static_cast<GLuint>( videoFrame.handle().toInt() );
        QOpenGLContext* ctx = QOpenGLContext::currentContext();
        QOpenGLFunctions* f = ctx->functions();
        GLuint fbo;
        f->glGenFramebuffers( 1, &fbo );
        GLint prevFbo;
        f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo );
        f->glBindFramebuffer( GL_FRAMEBUFFER, fbo );
        f->glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,  GL_TEXTURE_2D, textureId, 0 );
        f->glReadPixels( 0, 0,  videoFrame.width(),  videoFrame.height(), GL_RGBA, GL_UNSIGNED_BYTE, image.bits() );
        f->glBindFramebuffer( GL_FRAMEBUFFER, static_cast<GLuint>( prevFbo ) );
        return image.rgbSwapped();
    }
    return QImage();
}

I have a complete working app on GitHub that includes an implementation of the above function:

https://github.com/stephenquan/MyVideoFilterApp/blob/master/QVideoFrameToQImage.cpp

Harpole

Answer 4

For anyone ending up here after finding that the qt private function "qt_imageFromVideoFrame" no longer exists in Qt5.15.0, you can now use the newly implemented method QImage QVideoFrame::image().

QImage img = frame.image();

I've tested this on Ubuntu and it works.

For Ubuntu version 6.4 use: QImage img = frame.toImage();

Zebra

Thanks, you save life, hope it works on android too – Designer