Stream video from Raspberry pi to Opencv application on remote computer

Question

What I am trying to achieve is, I have a raspberry 3 with pi camera v2 connected to my local wifi. I want to transmit a live video from the raspberry pi to a computer running Ubuntu. On my computer I am trying to process that video with opencv in real time. The code below is just a sample code to test the video coming from the raspberry pi on my Ubuntu computer. I am using netcat to stream the video and I have listed the shell script below the code.

#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
using namespace std;
using namespace cv;

int main(int argc, char** argv)
{


    VideoCapture cap;
    cap.open("/dev/stdin");
    if (!cap.isOpened())
        return -1;

    Mat edges;
    namedWindow("edges",1);
    for(;;){
        Mat frame;
        cap>>frame;
        imshow("edges",frame);
        if(waitKey(30)>=0)break;
    }   

    return 0;
}

This is the code to play the stream in opencv.

1. First I redirect the stream to my opencv app using. nc -l -p 5001 | ./app
2. then run I the raspberry camera and stream it using netcat raspivid -t 999999 -o | nc x.x.x.x 5001 x being the client pc ip address.

This doesn't work for me but I have tried it with mplayer by running nc -l -p 5001 | mplayer -fps 31 -cache 1024 - and it works perfectly.

I think my problem is I am not capturing the stream properly on my opencv application. I need help please.

Jestinejesting

Please try and be a little clearer on what you are actually trying to achieve, and which code is running where, and how you compile it etc. Ad it stands, your code is incomplete - no header files at least. What camera are you using? What is your network like? What OS is running on each computer? – Kersten

I think I have made it more clearer. If you can offer any suggestion it would be appreciated. – Jestinejesting

I would run raspivid on your Raspberry Pi and capture 5 seconds of video into a local disk file on the Raspberry Pi. Then transfer the file to your Ubuntu using FTP, or scp and run your program with ./app < RecordedFile. If that works, the problem is either netcat, your network or your firewall. If it doesn't, it probably means OpenCV is not expecting the data in the format that the Raspberry Pi is delivering. Let us know how it goes! – Kersten

I transferred the video to ubuntu like you said and opencv seems to support h264. So the problem is not the format of the data. It's also not netcat because it works with mplayer. I also tried saving the stream into a file on ubuntu and opening that file in the opencv application. This works but the delay is too much 10-15s. – Jestinejesting

Answer 1

I spent around 4 hours trying to make this work and got it working in the end. I am not certain which was the key step, nor why, but I found it all works if I install the following packages on my Ubuntu 16 LTS VirtualBox:

libtbb2
ffmpeg
libavcodec
libavcodec-dev
libavformat-dev
libtbb-dev
libswscale-dev
libgtk2.0-dev
libtbb-dev
libjpeg-dev
libpng-dev
libtiff-dev
libjasper-dev
libdc1394-22-dev
libv4l-dev
libx264-dev

and run the camera on my Raspberry Pi with:

raspivid -ih -w 1024 -h 768 -o - | nc 192.168.0.98 5001

Keywords: Raspberry Pi, RASPI, raspivid, camera, OpenCV, stream, streaming, netcat, nc, network, video

Kersten

That worked perfectly. If you add the -t 0 param it will stream indefinitely. Thank you – Jestinejesting

I have used this project once, and it helps a lot github.com/mpromonet/v4l2rtspserver and on OpenCV you can just open the stream passing the URL like you do on any rtsp URL. To see if the RTSP is working properly you can use the VLC, I have done it using these tools. – Eldoree