OpenCV/FFMpeg image capture problems

Question

I'm trying to capture images from an IP camera in real time. The stream works perfectly well in VLC, but OpenCV's cvQueryFrame() seems to jumble and corrupt the incoming images to the point of no recognition.

Again, capturing from file works fine, but not a live stream. In case it makes a difference, I'm using an rtsp connection URL; I've also tried this with two different camera models (different brands), and the problem remains.

Besides, the (I'm assuming) codec is outputting several errors of the following kind: Error at MB: 1746 and concealing 6000 DC, 6000 AC, 6000 MV errors.

What can I do?

Update: The first error in the sequence is always cannot parallelize deblocking type 1, decoding such frames in sequential order

Update 2: Alright, it seems that OpenCV/FFMPEG has an issue with rtsp/h264 streams. I've tried the Qt Phonon library, which also doesn't work, and I've given the Live555 library a quick overview. This last appears to work, in the sense that everyone says it does, and the application example (OpenRTSP) in fact plays my stream well. However, to be quite honest, getting to grips with the Live555 code seems like a lengthy affair which I can hardly afford right now. Barring any other alternative, I guess I'll have to go that route.

Is there any other solution that comes to mind?

Update 3: I got the test RTSP client from the Live555 code to work, so I know how to extract h264 frame information from a stream, but now I need to recombine that frame information into actual displayable frames, which doesn't seem like something straightforward! Anyone familiar with Live555 know how to do this? Thanks.

Metallo

What version of OpenCV are you using? in it on linux or windows? – Claresta

cvQueryFrame() might return a NULL image. Be sure to test this before doing something with it. – Utrecht

Yes Karl, I'm already testing for null. – Piano

Why not converting the stream realtime and catch a different one from OpenCV? You could use gstreamer for that. I agree this is far from perfect, but you wouldn't have to touch Live555 – Lax

@KristianD'Amato Is anybody even checking this question now? – Fioritura

I am, believe you me. The below (till now, that is) solutions do not work. I've tried all basic approaches through OpenCV, and the problem is not in OpenCV, but in FFMpeg. ffplay does not open my camera's stream without corrupted images, so I need a replacement of FFMpeg/OpenCV. – Piano

I haven't use OpenCV, but it seems (from your comments) that problem lies in ffmpeg. Have you tried to use other version of ffmpeg? – Worried

Yes, I've tried the latest head as well as substantially older versions. The problem is not limited to my case, from what I can gather. – Piano

If it woked with VLC, why not use libvlc? It is high level and well documented. There is a "videodisplay" callback which is called for each frame which gives you image data so you can do whatever you want with it. – Pollux

Have you tried OpenCV 2.4.2 ? – Flounder

Can you please post your Live555 code? - did you ever get it working with OpenCV? -- My problem is there is no way to pass -rtsp_transport tcp to the opencv highgui video functions so the stream is requested with UDP, this is what causes all the problems. If you try to use ffplay without -rtsp_transport tcp, you get the same results, but specifying the tcp transport fixed the problem. Anyone found a solution to this? – Pastis

Has there been any other resolution to this problem? I'm experiencing a similar problem except that I get a reliable stream through ffplay but a distorted stream with OpenCV/FFMPEG (screen goes streaky at random intervals). I have tried recompiling the opencv source with the latest version of ffmpeg however this is proving difficult to do, and I doubt the version with the source which is 2 months old is broken (I'm using 2.4.8). I also see a delay of up to 20 seconds using opencv/ffmpeg which isn't acceptable for my use, and no way to specify ffmpeg params. I'm stuck but will try libvlc. – Laritalariviere

Answer 1

It seems you need an extra software layer to capture the stream packets and reconstruct the frames locally, and then feed them to openCV. You can easily achieve this with libVLC. This would also avoid codec problems since you can parse almost all codecs with libVLC and then feed raw frames to openCV.

Rabjohn

I am working on a similar issue. Could you please post code for feeding raw frames from libVLC to openCV? – Gimmal

Answer 2

I don't know if it helps (since I'm not an experienced c++ dev), but I've recently managed to get a stream from an IP Camera. Here's a quick test:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
#include <stdio.h>

using namespace cv;
using namespace std;

int main(int, char**)
{
    VideoCapture ipCam;
    Mat frame;
    const string LOCATION = "rtsp://192.168.0.200:554/rtsph264vga";

    if(!ipCam.open(LOCATION)) {
        cout << "Error opening video stream or file" << endl;
        return -1;
    }

    for(;;) {
        if(!ipCam.read(frame)) {
            cout << "No frame" << endl;
            waitKey();
        }
        imshow("cam", frame);
        if(waitKey(1) >= 0) break;
    }

    return 0;
}

Before heading to c++ I've setup the camera to export to H264 VGA (as it wasn't enabled by default on the cam I'm working with) and made sure I've got the stream running in VLC. I'm using OpenCV 2.4.1 with fffmpeg enabled. As far I understand the ffmpeg integration with OpenCV is available from OpenCV 2.0 upwards.

I did run into a few issues when I had to integrate merge the cv code with other c++ code as I have OpenCV and ffmpeg + dependencies built for 64bit arch. and the other code was relying on many 32bit libraries. The VideoCapture class is part of the highgui lib and that mainly the one you need to worry about. If it's not compiled with ffmpeg support you will get an error or a warning as VideoCapture won't be able to transcode the content.

Not sure it it's the best option, but you could try to stream/transcode the stream from VLC (by ticking Streaming/Saving in the Open Source/Network tab)

Corwin

Answer 3

Here is a code snippet, that I used to capture frames from WebCam. It worked for me, Hope it works for you as well...

int main(int argc, char* argv[])
{
    CvCapture *capture = NULL;
    IplImage* frame=NULL;
    int key =0;
    capture = cvCaptureFromCAM(0);

    if (!capture)   {
        printf("Cannot initailize webcam");
        return 1;
    }

    cvNamedWindow("result",CV_WINDOW_AUTOSIZE);

    while(key != 'q')
    {
        frame=cvQueryFrame(capture);

        if(!frame) break;

        cvShowImage("result",frame);
        key=cvWaitKey(10);
        frame=NULL;
    }
    cvDestroyWindow("result");
    cvReleaseCapture(&capture);
    return 0;
}

Cryptanalysis

As such, the problem is not in OpenCV, but the underlying FFMpeg libraries it uses. This does not work. – Piano

Answer 4

For OpenCV 2.3.1 I have written this code and it works normally, that is, I get the images from the camera feed.

VideoCapture cap(0);
if(!cap.isOpened())
{
    cout<<"Camera is not connected"<<endl;
    getchar();
} 
namedWindow("Camera Feed",1);
for(;;)
{
    Mat frame;
    cap >> frame;
    imshow("Camera Feed", frame);
    if(!frame.empty())
        detectAndDisplay(frame);
    else
        cout<<"No frame as input"<<endl;
    int c=waitKey(10);
    if(c==27)
        break;
}
return 0;

As you can see, it takes the input and displays it continuously and exits if you press ESC on your keyboard. Here's the documentation for CV 2.1 which has the same set of commands as CV 2.3. The commands changed from 2.4 I guess, although I am not too sure about it. Hope it helps.:)

Fioritura

Well, I've always managed to get the pictures from the cameras, even in this case, although in this case the images are corrupted. This is the C++ version of getting a video capture in OpenCV, but the result is the same because it wraps the same FFMpeg functionality. – Piano

@KristianD'Amato This is all I have. Images are corrupt? That does not happen. – Fioritura