How to parse mjpeg http stream from ip camera?

Question

Given below is the code written for getting live stream from an IP Camera.

from cv2 import *
from cv2 import cv
import urllib
import numpy as np
k=0
capture=cv.CaptureFromFile("http://IPADDRESS of the camera/axis-cgi/mjpg/video.cgi")
namedWindow("Display",1)

while True:
    frame=cv.QueryFrame(capture)
    if frame is None:
        print 'Cam not found'
        break
    else:
        cv.ShowImage("Display", frame)
    if k==0x1b:
        print 'Esc. Exiting'
        break

On running the code the output that I am getting is:

Cam not found

Where am I going wrong? Also, why is frame None here? Is there some problem with the conversion?

Humber

Is that CGI script returning the video stream or a HTML page for browser display? – Misbelief

@Misbelief It returns a video stream, I have tried playing it using VLC and it works. – Humber

I have no IP camera but others have fought with Axis cameras a lot in 2009. Apart from that "mjpg" at the end of the URL may help. – Misbelief

As I see you have tried different aproaches as well and fixed the mjpg already. :) One last guess is using gstreamer between OpenCV and the IP camera. – Misbelief

@Misbelief I saw that, see my comment on that answer. – Humber

What is gstreamer? And how do I use it? Any tutorials/guides/questions on it? In the meantime I'll try installing it on windows. – Humber

Please check this link: xuv.be/OpenCV-GStreamer-Camera-over-IP.html (although it's Processing, not python). – Misbelief

@Misbelief Ah, I tried processing, the problem is it does not DB support and also, it can primarily be used for Data visualization and I cannot use it to build a complete application. As in cannot add button, text boxes etc. etc. Although I guess libraries must be present but I could not find any which could work on all three (windows, linux and iOS). If you do know then tell me, I shall try. – Humber

I'm using VideoCapture("url/to/cam") and that works just fine. – Isabellisabella

Answer 1

import cv2
import urllib 
import numpy as np

stream = urllib.urlopen('http://localhost:8080/frame.mjpg')
bytes = ''
while True:
    bytes += stream.read(1024)
    a = bytes.find('\xff\xd8')
    b = bytes.find('\xff\xd9')
    if a != -1 and b != -1:
        jpg = bytes[a:b+2]
        bytes = bytes[b+2:]
        i = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.CV_LOAD_IMAGE_COLOR)
        cv2.imshow('i', i)
        if cv2.waitKey(1) == 27:
            exit(0)   

edit (explanation)

I just saw that you mention that you have c++ code that is working, if that is the case your camera may work in python as well. The code above manually parses the mjpeg stream without relying on opencv, since in some of my projects the url will not be opened by opencv no matter what I did(c++,python).

Mjpeg over http is multipart/x-mixed-replace with boundary frame info and jpeg data is just sent in binary. So you don't really need to care about http protocol headers. All jpeg frames start with marker 0xff 0xd8 and end with 0xff 0xd9. So the code above extracts such frames from the http stream and decodes them one by one. like below.

...(http)
0xff 0xd8      --|
[jpeg data]      |--this part is extracted and decoded
0xff 0xd9      --|
...(http)
0xff 0xd8      --|
[jpeg data]      |--this part is extracted and decoded
0xff 0xd9      --|
...(http)

edit 2 (reading from mjpg file)

Regarding your question of saving the file, yes the file can be directly saved and reopened using the same method with very small modification. For example you would do curl http://IPCAM > output.mjpg and then change the line stream=urllib.urlopen('http://localhost:8080/frame.mjpg')so that the code becomes this

import cv2
import urllib 
import numpy as np

stream = open('output.mjpg', 'rb')
bytes = ''
while True:
    bytes += stream.read(1024)
    a = bytes.find('\xff\xd8')
    b = bytes.find('\xff\xd9')
    if a != -1 and b != -1:
        jpg = bytes[a:b+2]
        bytes = bytes[b+2:]
        i = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.CV_LOAD_IMAGE_COLOR)
        cv2.imshow('i', i)
        if cv2.waitKey(1) == 27:
            exit(0)   

Of course you are saving a lot of redundant http headers, which you might want to strip away. Or if you have extra cpu power, maybe just encode to h264 first. But if the camera is adding some meta data to http header frames such as channel, timestamp, etc. Then it may be useful to keep them.

edit 3 (tkinter interfacing)

import cv2
import urllib 
import numpy as np
import Tkinter
from PIL import Image, ImageTk
import threading

root = Tkinter.Tk()
image_label = Tkinter.Label(root)  
image_label.pack()

def cvloop():    
    stream=open('output.mjpg', 'rb')
    bytes = ''
    while True:
        bytes += stream.read(1024)
        a = bytes.find('\xff\xd8')
        b = bytes.find('\xff\xd9')
        if a != -1 and b != -1:
            jpg = bytes[a:b+2]
            bytes = bytes[b+2:]
            i = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.CV_LOAD_IMAGE_COLOR)            
            tki = ImageTk.PhotoImage(Image.fromarray(cv2.cvtColor(i, cv2.COLOR_BGR2RGB)))
            image_label.configure(image=tki)                
            image_label._backbuffer_ = tki  #avoid flicker caused by premature gc
            cv2.imshow('i', i)
        if cv2.waitKey(1) == 27:
            exit(0)  

thread = threading.Thread(target=cvloop)
thread.start()
root.mainloop()

Brechtel

bytes is a growing queue that is slowly consumed by the parsing loop, the stream.read(16384) read 16384 bytes of data from http stream and add it to bytes,which will be shortened if a valid jpeg frame is found. the read buffer should be smaller than the smallest jpeg frame size, otherwise, there might be problems. – Brechtel

btw, i have also needed to do the same in c++, which is posted here(#20900011). idea is the same, but implementation there is more robust than python and is production tested. might be useful for you if you want to maintain same behaviour for both of your codebase – Brechtel

Ah, I was just going to ask you for a similar C++ code. :) So another question, the bytes that are read here from the stream contain images in binary format? If so, can I save this stream of data? Then I am able to view it later? – Humber

Do you know why it would lag? The more the duration of the video the more the lag? – Humber

Hmm...there shouldnt be any lag. Are you talking about the python version or c++ version? Can you describe in more details? – Brechtel

Ah, I figured it out. Although I don't know why but the python version was lagging more and more when I was running it. Then I changed the size from 16384 to 1024 and it is running fine. Don't know why it was lagging and why its working completely fine now. – Humber

i think it can slightly increased cpu usage. the read buffer shouldnt be too small since, find operation if repeatedly done too many times can be quite expensive. to be honest, python version was a lazy(but easy) implementation just to quickly test for experimental work. maybe if i have some time during this weekend, i can update with a slightly better method. – Brechtel

Awaiting the updated answer if you do update. :) Thanks again! – Humber

Okay another question, what do you use jpg = bytes[a:b+2] bytes= bytes[b+2:] for? – Humber

Okay another question, what do you use jpg = bytes[a:b+2] bytes= bytes[b+2:] for? I mean i get that it is extracting the jpg part from the stream again and again but why b+2? – Humber

just bytes[a:b] without b+2 will not contain '\xff\xd9'. so you need to add offset of 2 to accommodate for 2 bytes of end markers. – Brechtel

And lastly, can I display the jpg image using Tkinter itself? Not opencv? – Humber

hmm...i got curious about how to do this as well. i have edited the answer. i know you got it solved in another question, but i think this method is better because the gui is going to hang in io errors etc in that answer. – Brechtel

Geez! you are the computer hacker! – Throw

Any chance you could port this to Python 3? Strings and bytes and all that make this broken and I was unable to adapt it myself. – Frowzy

Ported it myself, posted as another answer. Thanks! – Frowzy

Hi Zan Lin. Thank for the answer. In my case it has the following error: IOError: ('http error', 401, 'Unauthorized', <httplib.HTTPMessage instance at 0x7f733cbc57e8>). How would I modify to allow for authorization? :) – Rhearheba

What type of authentication scheme? If it's basic, following something along the line of (#635613) should work. For other types, I have seen people recommending request library (docs.python-requests.org/en/latest/user/authentication/…). I have not tried any one of them though. But the only thing you need to change is the stream=urllib... maybe it will become 4 lines instead of one line. – Brechtel

I get an error ... TypeError: 'bytes' object is not callable – Heraldry

This is wrong solution, it generates broken frames. Because these magic numbers may present in the middle of JPEG data, so this solution will work by chance, most of frames will be ok, but some will be missing. This is correct way of doing this, in the bottom of the link: github.com/nodejs/node/issues/7533 (you should use HTTP delimiter, which comes with the header) – Semen

As far as I know, the end markers cannot appear inside JPEG data unless there are jpegs embedded inside jpegs, in which case, the issue will happen for all frames and fairly easy to know that it is failing. – Brechtel

@Frowzy You can try to modify below code, it is ok for me in python3 Before bytes = bytes(1024) After bytes = b"" – Mylo

Answer 2

First of all, please be aware that you should first try simply using OpenCV's video capture functions directly, e.g. cv2.VideoCapture('http://localhost:8080/frame.mjpg')!

This works just fine for me:

import cv2
cap = cv2.VideoCapture('http://localhost:8080/frame.mjpg')

while True:
  ret, frame = cap.read()
  cv2.imshow('Video', frame)

  if cv2.waitKey(1) == 27:
    exit(0)

Anyways, here is Zaw Lin's solution ported to OpenCV 3 (only change is cv2.CV_LOAD_IMAGE_COLOR to cv2.IMREAD_COLOR and Python 3 (string vs byte handling changed plus urllib):

import cv2
import urllib.request
import numpy as np

stream = urllib.request.urlopen('http://localhost:8080/frame.mjpg')
bytes = bytes()
while True:
    bytes += stream.read(1024)
    a = bytes.find(b'\xff\xd8')
    b = bytes.find(b'\xff\xd9')
    if a != -1 and b != -1:
        jpg = bytes[a:b+2]
        bytes = bytes[b+2:]
        i = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
        cv2.imshow('i', i)
        if cv2.waitKey(1) == 27:
            exit(0)

Frowzy

If you're trying to adapt the accepted answer to use python3's urllib, beware of changing the bytes.find('...') to bytes.find(b'...') - basically remember to specify that we're looking for byte! Otherwise won't work – Humdrum

I get following error bytes = bytes() TypeError: 'str' object is not callable – Heraldry

@Heraldry I cannot reproduce that statement raising a TypeError in neither 2.7.15 nor 3.7.0. – Frowzy

Thanks for trying I used cap = cv2.VideoCapture('http:/5102.54/video/mjpeg/stream2') and it works. – Heraldry

It comes out that both work. Obviously the VideoCapture is shorter. – Hurty

Shorter but also way more robust as you rely on the well tested code of others instead of trying to parse bits and bytes yourself. ;) – Frowzy

Make sure you have the video url and not the snapshot url (like me) :) – Talich

Answer 3

Here is an answer using the Python 3 requests module instead of urllib.

The reason for not using urllib is that it cannot correctly interpret a URL like http://user:pass@ipaddress:port

Adding authentication parameters is more complex in urllib than the requests module.

Here is a nice, concise solution using the requests module:

import cv2
import requests
import numpy as np

r = requests.get('http://192.168.1.xx/mjpeg.cgi', auth=('user', 'password'), stream=True)
if(r.status_code == 200):
    bytes = bytes()
    for chunk in r.iter_content(chunk_size=1024):
        bytes += chunk
        a = bytes.find(b'\xff\xd8')
        b = bytes.find(b'\xff\xd9')
        if a != -1 and b != -1:
            jpg = bytes[a:b+2]
            bytes = bytes[b+2:]
            i = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
            cv2.imshow('i', i)
            if cv2.waitKey(1) == 27:
                exit(0)
else:
    print("Received unexpected status code {}".format(r.status_code))

Mendelevium

Varun, thanks for the answer. I am wondering if you'd know a way to use the same generator, that requests provides but somehow "return" that decoded image? Or to put it in other words, extract it outside of this part of code? I asked another answer but I don't expect it to get enough attention (#44157660). – Rainstorm

Defining a variable with name bytes will shadow the builtin bytes which is a type in Python 3. Better use another name. – Rogue

It seems that with the recent version of DroidCam app, urlopen call fails to fetch the stream. This code snippet using requests module works like a charm! – Sicklebill

Answer 4

I don't think the first anwser is fine with other format image data, eg png. So I write the following code, which can handle other type of images

"""
MJPEG format

Content-Type: multipart/x-mixed-replace; boundary=--BoundaryString
--BoundaryString
Content-type: image/jpg
Content-Length: 12390

... image-data here ...


--BoundaryString
Content-type: image/jpg
Content-Length: 12390

... image-data here ...
"""
import io
import requests
import cv2
import numpy as np


class MjpegReader():
    def __init__(self, url: str):
        self._url = url

    def iter_content(self):
        """
        Raises:
            RuntimeError
        """
        r = requests.get(self._url, stream=True)

        # parse boundary
        content_type = r.headers['content-type']
        index = content_type.rfind("boundary=")
        assert index != 1
        boundary = content_type[index+len("boundary="):] + "\r\n"
        boundary = boundary.encode('utf-8')

        rd = io.BufferedReader(r.raw)
        while True:
            self._skip_to_boundary(rd, boundary)
            length = self._parse_length(rd)
            yield rd.read(length)

    def _parse_length(self, rd) -> int:
        length = 0
        while True:
            line = rd.readline()
            if line == b'\r\n':
                return length
            if line.startswith(b"Content-Length"):
                length = int(line.decode('utf-8').split(": ")[1])
                assert length > 0


    def _skip_to_boundary(self, rd, boundary: bytes):
        for _ in range(10):
            if boundary in rd.readline():
                break
        else:
            raise RuntimeError("Boundary not detected:", boundary)

mr = MjpegReader("http://127.0.0.1/mjpeg.cgi")
for content in mr.iter_content():
    i = cv2.imdecode(np.frombuffer(content, dtype=np.uint8), cv2.IMREAD_COLOR)
    cv2.imshow('i', i)
    if cv2.waitKey(1) == 27:
        break

Oppress

Answer 5

I had the same problem. The solution without requests or urllib: just add the user and password in the cam address, using VideoCapture, like this:

E.g.

cv2.VideoCapture('http://user:[email protected]/video')

using IPWebcam for android.

Bradleybradly