opencv single h264 raw frame as a binary string

Question

have created a rtsp client in python that receives a h264 stream and returns single h264 raw frames as a binary strings. I am trying to process each h264 frames on-the-fly.

I have unsuccessfully tried several ways to convert this frame into a numpy array for processing.

So far I know that cv2.VideoCapture only accepts a file name as it argument, not a frame neither a StringIO object (file like pointer to a buffer), but I need to pass to it my string.

I have also tried something like:

nparr = np.fromstring(frame_bin_str, np.uint8)
img_np = cv2.imdecode(nparr, cv2.CV_LOAD_IMAGE_COLOR)

tried diferent flags. but also failed miserably.

after many other failed attempts , I ran out of ideas.

To summarize what I need to do: I have a h264 raw frame in a variable and I need to create an openvc valid numpy array of it, or somehow end up with a VideoCapture object containing that single frame, so I can process the frame.

Any pointers would be much appreciated.

Hope this all makes sense.

Thank you in advance

Felice

afaik openCV doesn't support h264 raw format. In our current project someone wrote some code which reads those files frame by frame (jumping to byte positions for a given frame number) and interpretes byte information manually to get openCV usable data according to h264 format rules. – Jaddan

Thank you for your response Micka. If I dump the frames into a file with a "\x0\x00\x00\x001" separator. and I create a VideoCapture object from it. all works just perfect. I can read and retrieve frame by frame and manipulate them fine. So I assumed the support was there. I can't figure how to load a single frame rather than a file. – Felice

did you ever find a way to do this? – Changchangaris

Answer 1

As Micka suggested, there is no support for h264 RAW format in OpenCV and we should convert it ourselves.

I think you should be reshaping the nparr to the shape of the incoming image. Not necessary to do imdecode. Use imshow to display the result and verify.

Here is the code I used to convert a 16 bit RAW image (grayscale) in a similar way. I have renormalized my image before displaying.

framenp = np.fromstring(framestr, dtype=np.uint16).reshape((1024,1280))
#renormalizing to float
framenp = (framenp*1./framenp.max())
framenp.dtype = np.float
cv2.imshow('frame', cv2.resize(framenp, (640,480)))

Ichnology