Python OpenCV streaming from camera - multithreading, timestamps

Question

I ran simple python script on Raspberry Pi 3. This script is responsible to open video device and stream data (800x600) to HTTP endpoint using MJPEG. When I receive this stream one of my Raspberry Pi cores works on 100%. It possible to run OpenCV with multi threading?

This is my code

import cv2
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import time
import argparse
import socket as Socket    
camera = None  

def setUpCameraCV():
    global camera
    camera = cv2.VideoCapture(0)

class mjpgServer(BaseHTTPRequestHandler):

    ip = None
    hostname = None

    def do_GET(self):

        print('connection from:', self.address_string())

        if self.ip is None or self.hostname is None:
            self.ip, _ = 0.0.0.0
            self.hostname = Socket.gethostname()

        if self.path == '/mjpg':

            self.send_response(200)
            self.send_header('Cache-Control', 'no-cache')
            self.send_header('Pragma', 'no-cache')
            self.send_header('Connection', 'close')
            self.send_header(
                'Content-type',
                'multipart/x-mixed-replace; boundary=mjpegstream'
            )
            self.end_headers()

            while True:
                if camera:
                    ret, img = camera.read()

                else:
                    raise Exception('Error, camera not setup')

                if not ret:
                    print('no image from camera')
                    time.sleep(1)
                    continue

                ret, jpg = cv2.imencode('.jpg', img)
                
                self.end_headers()
                self.wfile.write('--mjpegstream')
                self.end_headers()

                self.send_header('Content-type', 'image/jpeg')
                self.send_header('Content-length', str(jpg.size))
                self.end_headers()
                self.wfile.write(jpg.tostring())    

def main():
    try:
        setUpCameraCV()         
        mjpgServer.ip = 0.0.0.0
        mjpgServer.hostname = Socket.gethostname()
        server = HTTPServer((ipv4, args['port']), mjpgServer)
        print("server started on {}:{}".format(Socket.gethostname(), args['port']))
        server.serve_forever()

    except KeyboardInterrupt:
        print('KeyboardInterrupt')

    server.socket.close()


if __name__ == '__main__':
    main()

Another question, how to get timestamp of each frame on the client side (receiver) it possible?

Carnarvon

Take a look here – Brooklime

Answer 1

Using threading to handle I/O heavy operations (such as reading frames from a webcam) is a classic programming model. Since accessing the webcam/camera using cv2.VideoCapture().read() is a blocking operation, our main program is stalled until the frame is read from the camera device and returned to our script. Essentially the idea is to spawn another thread to handle grabbing the frames in parallel instead of relying on a single thread (our 'main' thread) to grab the frames in sequential order. This will allow frames to be continuously read from the I/O thread, while our root thread processes the current frame. Once the root thread finishes processing its frame, it simply needs to grab the current frame from the I/O thread without having to wait for blocking I/O operations.

Thus we can improve performance by creating a new thread that does nothing but poll for new frames while our main thread handles processing the current frame. For an implementation to handle multiple camera streams, take a look at capture multiple camera streams with OpenCV

from threading import Thread
import cv2, time
 
class VideoStreamWidget(object):
    def __init__(self, src=0):
        self.capture = cv2.VideoCapture(src)
        # Start the thread to read frames from the video stream
        self.thread = Thread(target=self.update, args=())
        self.thread.daemon = True
        self.thread.start()

    def update(self):
        # Read the next frame from the stream in a different thread
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()
            time.sleep(.01)
    
    def show_frame(self):
        # Display frames in main program
        cv2.imshow('frame', self.frame)
        key = cv2.waitKey(1)
        if key == ord('q'):
            self.capture.release()
            cv2.destroyAllWindows()
            exit(1)

if __name__ == '__main__':
    video_stream_widget = VideoStreamWidget()
    while True:
        try:
            video_stream_widget.show_frame()
        except AttributeError:
            pass

---

Related camera/IP/RTSP, FPS, video, threading, and multiprocessing posts

1. Python OpenCV streaming from camera - multithreading, timestamps

2. Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture

3. How to capture multiple camera streams with OpenCV?

4. OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?

5. Storing RTSP stream as video file with OpenCV VideoWriter

6. OpenCV video saving

7. Python OpenCV multiprocessing cv2.VideoCapture mp4

Aviatrix

Curious, i) Why is the time.sleep(.01) used? ii) Is it needed when reading from real IP cameras? Because the solution I'm working on needs real time, but this 100ms addition is a lot of delay when looked at, from a bigger picture. – Telpherage

@AakashBasu time.sleep is used because the camera can only poll so fast, there's no need to poll at a ridiculous rate if there's no new frames in the buffer. I highly recommend it, you would have to look at your IP camera specs. Most IP cameras are in the 30-60 FPS range so polling any faster will have no benefit. You can reduce the delay or take it out entirely, I found out that the delay was useful for my camera – Aviatrix

So a good point of reference would be 1 second / fps? e.g. on a 30 fps camera you should poll at 1/30 = 0.033 seconds or less? – Rapture

@Rapture Sounds about right, my cameras are only capable of 30 FPS. A good idea is to poll a bit faster then your camera's capacity to ensure that you always get the latest frame. Every situation is different so just experiment to determine what value produces smooth frames – Aviatrix

this code never checks self.status. further, the check for capture.isOpened() is wrong. it's supposed to be checked ONCE, BEFORE any loops are entered. this code simply causes an infinite loop instead. that is BAD. – Lakeishalakeland