Display on screen using QAbstractVideoSurface [closed]

Question

I am trying to display camera picture on screen using a subclass of QAbstractVideoSurface, and I've no experience on this.

I'd appreciate if anyone can explain how to do it.

Vasoconstrictor

A good question is supposed to indicate some level of research... For example, did you try to search for any associated documentation? The way it is stated, the question seems very open ended. – Tortricid

yes, I have been strugling for a month now. I did use cameraviewfinder, that only displays what camera sees. In order to get videoframes, I should probably use the method mentioned above. But there is lack of examples on the Internet I could try. thank u for your comment, this was my first question, so please don't be too strict – Vasoconstrictor

@Vasoconstrictor do you struggle with the subclass implementation, or how to use the VideoSurface object? – Dissert

as far as I understand, sublass is needed to get frames. What is unclear to me is displaying video on screen. – Vasoconstrictor

Answer 1

The QAbstractVideoSurface is an interface between the producer and consumer of the video frames. You only have two functions to implement to begin with:

1. supportedPixelFormats so that the producer can select an appropriate format for the QVideoFrame
2. present which is more generic wording for show\display this frame

Lets say you want to use a classic QWidget for the display. In this case, you may choose to use a QImage to draw on the widget.

First Qt is guaranteed to paint a QImage which is RGB24 (or BGR24) on most platforms. So

QList<QVideoFrame::PixelFormat> LabelBasedVideoSurface::supportedPixelFormats(
        QAbstractVideoBuffer::HandleType handleType) const
{
    if (handleType == QAbstractVideoBuffer::NoHandle) {
        return QList<QVideoFrame::PixelFormat>()
                << QVideoFrame::Format_RGB24;
    } else {
        return QList<QVideoFrame::PixelFormat>();
    }
}

Now to present the QVideoFrame, you map its data to a QImage, and paint the QImage to the widget. For simplicity I will use a QLabel, that I access directly (no signal no slot).

bool LabelBasedVideoSurface::present(const QVideoFrame &frame)
{
    if (notMyFormat(frame.pixelFormat())) {
        setError(IncorrectFormatError);
        return false;
    } else {

        QVideoFrame frametodraw(frame);

        if(!frametodraw.map(QAbstractVideoBuffer::ReadOnly))
        {
           setError(ResourceError);
           return false;
        } 

         //this is a shallow operation. it just refer the frame buffer
         QImage image(
                frametodraw.bits(),
                frametodraw.width(),
                frametodraw.height(),
                frametodraw.bytesPerLine(),
                QImage::Format_RGB444);

        mylabel->resize(image.size());

        //QPixmap::fromImage create a new buffer for the pixmap
        mylabel->setPixmap(QPixmap::fromImage(image));

        //we can release the data
        frametodraw.unmap();

        mylabel->update();

        return true;
    }
}

This example is obviously not optimal.

1. It doesnt take cash on the fact that a QVideoFrame might be stored in video memory, because we are drawing using a pixmap.
2. The conversion from image to pixmap is a unnecessary hit.

You can write your own widget, and implement a paintEvent for better performance. Also, you have several design liberties on how present() behave. For instance :

• Whether is a non blocking surface, ie the frame is already shown when present finishes. Above it will mean using mylabel->repaint() instead of mylabel->update()
• What happens when you cannot complete the presentation. You may want to draw a blank frame rather than returning an error who may stop the music.

Dissert

thank you so much, this helped me a lot! However, the image shown on display is green and magenta, any suggestions how to fix it? – Vasoconstrictor

QVideoFrame::Format_RGB24 might be QVideoFrame::Format_BGR24. Play with the different rgb formats you have to match your source. – Dissert

What thing is going to call present()? I added this class to my QtMediaPlayer but QtMediaPlayer won't call present(), ever. – Conoscenti