TCP or UDP? Delays building up on production for video stream

Question

I am creating a video stream from a camera with FFMPEG and nodejs stream Duplex class.

this.ffmpegProcess = spawn('"ffmpeg"', [

  '-i', '-',
  '-loglevel', 'info',

  /**
   * MJPEG Stream
   */

  '-map', '0:v',
  '-c:v', 'mjpeg',
  '-thread_type', 'frame', // suggested for performance on StackOverflow.
  '-q:v', '20', // force quality of image. 2-31 when 2=best, 31=worst
  '-r', '25', // force framerate
  '-f', 'mjpeg',
  `-`,

], {
  shell: true,
  detached: false,
});

On local network, we are testing it with a few computers and every thing works very stable with latency of maximum 2 seconds.

However, we have imported the service to AWS production, when we connect the first computer we have a latency of around 2 seconds, but if another client connects to the stream, they both start delaying a lot, where the latency builds up to 10+ seconds, and in addition the video is very slow, as in the motion between frames.

Now I asked about TCP or UDP is because we are using TCP for the stream, which means that every packet that is sent is implementing the TCP syn-ack-synack protocol and sequence.

My question is, could TCP really be causing such an issue that the delays get up to 10 seconds with very slow motion? because on local network it works very just fine.

Dune

Do you mean you have a TCP connect for each single packet? – Therontheropod

Answer 1

Yes, TCP is definitely not correct protocol for it. Can be used but with some modification on the source side. UDP sadly is not a magic bullet, without additional logic UDP will not solve the problem either (if you don't care that you will see broken frames constructed randomly from other frames).

Explanation

The main features of the TCP are that packets are delivered in the correct order and all packets are delivered. These two features are very useful but quite harming for video streaming.

On the local network bandwidth is quite large and also packet loss is very low so TCP works fine. On the internet bandwidth is limited and every reach of the limit causes packet loss. TCP will deal with packet loss by resending the packet. Each resend will cause extending delay of the stream.

For better understanding try to imagine that a packet contains whole frame (JPEG image). Let's assume that normal delay of the link is 100ms (time in which frame will be in transit). For 25 FPS you need to deliver each frame every 40ms. If the frame is the lost in transit, TCP will ensure that copy will resend. TCP can detect this situation and fix in ideal situation in 2 times of delay - 2*100ms (in the reality it will be more, but for sake of simplicity I am keeping this). So during image loss 5 frames are waiting in the receiver queue and waits for single missing frame. In this example one missing frame causes 5 frames of delay. And because TCP creates queue of packets. delay will not be ever reduced, only can grow. In the ideal situation when bandwidth is enough delay will be still the same.

How to fix it

What I have done in the nodejs was fixing the source side. Packets in the TCP can be skipped only if the source will do it, TCP has no way how to do it itself.

For that purpose, I used event drain. Idea behind the algorithm is that ffmpeg generates frame it's own speed. And node.js reads frames and always keep the last received. It has also outgoing buffer with size of a single frame. So if sending of single frame is delayed due to network conditions, incoming images from ffmpeg are silently discarded (this compensates low bandwidth) except the last received. So when TCP buffer signals (by drain event) that some frame was correctly sent, nodejs will take the last received image and write it to the stream.

This algorithm regulates itself. If sending bandwidth is enough (sending is faster then frames generated by ffmpeg) then no packet will be discarded and 25fps will be delivered. If bandwidth can transfer only half of frames in average one of two frames will be discarded so receiver will se 12.5fps but not growing delay.

Probably the most complicated part in this algorithm is correctly slicing byte stream to frames.

Rosalba

Thank you for your answer, we have implemented exactly what you said right after I posted this question, it did improve the performance a bit, but it's bad. things become slow and then sync again with fast-motion, I am not sure why. I assume that facebook/instagram live is a game of pre-buffering so thats why it seems to work very fast, but in reality you cant know their real latency. – Dune

@BenBeri Another possibility is to use WebRTC (UDP) + h.264 (assuming that browser is used to display the stream) - you will get better compression ratio and performance but I have no first hand experience with this technology. – Rosalba

Answer 2

On either the server or a client, run a wireshark trace, this will tell you the exact "delay" of the packets, or which side is not doing the right thing. It sound's like the server is not living up to your expectations. Make sure the stream is UDP.

Antitrades

Answer 3

This might not be relevant for your use case, but this is where a HTTP based content delivery system, like HLS, is very useful.

UDP or TCP, trying to stream the data in real time across the internet is hard.

HLS will spit out files that can be downloaded over HTTP (which is actually quite efficient when serving large files, and ensures the integrity of the video as it's got built in error correction and so on), and re-assembled in order on the client side by the player.

It adapts easily to different quality streams (part of the protocol) which lets the client decide how quickly it consume the data.

As an aside, Netflix and the like use Widevine DRM which is somewhat similar - HTTP based delivery of the content, although widevine will take the different quality encodes, stuff it into one gigantic file and use HTTP range requests to switch between quality streams.

Syck

You should look into what HLS actually is: en.wikipedia.org/wiki/HTTP_Live_Streaming HLS is not an optimal solution for live video, as it requires pre-processing of the video stream, which introduces delay. HLS uses TCP as its transport (HTTP is a TCP-based protocol). Widevine is not relevant to this discussion: it is an authentication and access-management system that works with a variety of transports. If there's a performance problem that's caused by using TCP, using HLS cannot address it. – Thomasthomasa