Write image frames into gstreamer rtp pipeline

Question

I am trying to use the gstreamer pipeline to view an rtp stream in vlc on my computer. I mostly looked into this thread. My end result is something like this

#!/usr/bin/env python

import gi
import numpy as np

gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject
import time


class RtpPipeline(object):
    def __init__(self):
        self.number_frames = 0
        self.fps = 30
        self.duration = 1 / self.fps * Gst.SECOND  # duration of a frame in nanoseconds
        self.launch_string = 'appsrc name=source ' \
                             '!videoconvert !x264enc speed-preset=ultrafast tune=zerolatency byte-stream=true ' \
                             '! mpegtsmux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5000'
        pipeline = Gst.parse_launch(self.launch_string)
        appsrc = pipeline.get_child_by_name('source')

        while True:
            try:
                img = np.zeros([320, 320, 3], dtype=np.uint8)
                img.fill(255)  # white image
                data = img.tostring()
                buf = Gst.Buffer.new_allocate(None, len(data), None)
                buf.fill(0, data)
                buf.duration = self.duration
                timestamp = self.number_frames * self.duration
                buf.pts = buf.dts = int(timestamp)
                buf.offset = timestamp
                self.number_frames += 1
                retval = appsrc.emit('push-buffer', buf)
                if retval != Gst.FlowReturn.OK:
                    print(retval)
                time.sleep(0.2)
            except Exception as e:
                break


Gst.init(None)

factory = RtpPipeline()

loop = GObject.MainLoop()
loop.run()

Which doesn't throw an error but does not show anything in my vlc client. Any tips would be great (OpenCV VideoWriter is not an option).

Lindeberg

Answer 1

I think the most important thing that was missing was:

pipeline.set_state(Gst.State.PLAYING)

I also added the comments from mail2subhajit. Here is the final result, which is kinda buggy but is a good start.

class RtpPipeline(object):
    def __init__(self):
        self.number_frames = 0
        self.fps = 30
        self.cap = cv2.VideoCapture(0)
        self.duration = 1 / self.fps * Gst.SECOND  # duration of a frame in nanoseconds
        self.launch_string = 'appsrc name=source is-live=true format=GST_FORMAT_TIME ' \
                             ' caps=video/x-raw,format=BGR,width=640,height=480,framerate=30/1 ' \
                             '! videoconvert ! video/x-raw,format=I420 ' \
                             '! x264enc speed-preset=ultrafast tune=zerolatency byte-stream=true ' \
                             '! mpegtsmux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5000 sync=false'
        pipeline = Gst.parse_launch(self.launch_string)
        appsrc = pipeline.get_child_by_name('source')
        pipeline.set_state(Gst.State.PLAYING)

        while True:
            try:
                ret, frame = self.cap.read()
                start = time.time()
                data = frame.tostring()
                buf = Gst.Buffer.new_allocate(None, len(data), None)
                buf.fill(0, data)
                buf.duration = self.duration
                timestamp = self.number_frames * self.duration
                buf.pts = buf.dts = int(timestamp)
                buf.offset = timestamp
                self.number_frames += 1
                retval = appsrc.emit('push-buffer', buf)
                if retval != Gst.FlowReturn.OK:
                    print(retval)
            except Exception as e:
                break

Lindeberg

Answer 2

The Image frames need to be decoded (based on the format you are reading the image file from ) and then converted to the RAW formats ( RGB/BGR/YUV etc.) and then passed to the gstreamer pipeline.

if you are filling the image buffer with specific color, and not reading from any image files, you should know what format of the image buffer.

The Gstreamer pipeline also should know what RAW format is being passed to it with what resolution and fps details.

Python script should push the image files at the same frame rate as set in the fps. check you sleep time.

Check after changing these parameters, if the problem still exists - add the gstreamer logs too.

Abukir