FMP4 moof box sequence number ordering

Question

I wanted to do a basic fragmented mp4 broadcast program with avformat libs and HTML5 video and MSE.

This is a live stream and I use avformat to copy h264 data to mp4 fragments.

Here is my basic drawing of clients attaching to the stream:

So, with words:

1. C1J: First Client joins:
   • avformat process starts
   • ftyp, moov, moof, mdat boxes will be served to Client1
   • ftyp and moov atoms are both saved for later reuse
2. C2J: Second Client joins (later in time):
   • avformat process is ongoing (because it is still serving moof and mdat boxes for Client1)
   • previously saved ftyp and moov boxes will be served first to Client2
   • after ftyp and moov boxes were served, Client2 will join to the stream at the next moof box.

I have saved an mp4 file to disk from both clients.

Atoms' order within both files looks good: ftype, moov, moof, mdat, moof, mdat...

Both files can be played by media players (like VLC) and also in browsers directly (Opera).

Client1 can be played also via MSE in the browser (Opera), but Client2's stream is not displaying with MSE (Opera).

No errors on the JS console, and media-internals looks also good (at least equivalent with Client1's one).

Now I realized that every moof box contains an mfhd box (header) with a sequenceNumber field.

Of course in Client1's first moof box this sequenceNumber is 1. However in the later joined Client2's first moof box this sequenceNumber is always >= 1 (in my case it is 16).

What do I need to modify in the moof boxes in Client2 to have a valid fmp4 from the beginning?

I think Opera's HTML5 video does not like if sequenceNumber does not start from 1, but there shall be other requirements for being it valid.

Ahrendt

“ I think Opera's HTML5 video does not like if sequenceNumber does not start from 1” This is not true. That restriction would make live streams, and seeking impossible. The problem is not the sequence number. Do all your segments begin with an IDR? – Lattimer

Yes, IDRs are in the mdats (but they are preceeding by pps and sps). – Ahrendt

Thus, Client1 is playing the stream even after Client2 joins (after which both are getting the very same fragments). – Ahrendt

Answer 1

The mode attribute of the SourceBuffer interface is used to control whether the order in which media fragments are added to the SourceBuffer can be arbitrary or strict. Just set it to "sequence"

sourceBuffer.mode = "sequence";

https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer/mode

Washable

Answer 2

Once you have buffered some video, you have to set currentTime on the video element in JS to the latest buffered.end(0) time (assuming you have a single buffered range).

The problem is that the video you are playing is interpreted by the browser as starting in time zero, but you are feeding MSE fragments that are for later times.

Worsted

Answer 3

All moof boxes contain information about the order in which they should be played. MSE expects expects by default, the video to start at time 0.

One simple solution to this problem is setting the HTMLMediaElement.currentTime property to high value after you append the first buffer. Alternatively, to support MSE spec, the very first moof box you feed to the MSE buffer, should have first_sample_flags_present flag present.

Handbreadth

Flag name is first_sample_flags_present, not first_sample_flags_preset, that last one doesn't exists. I wanted to edit your answer, but I have edits queue full, so SO doesn't allows me to do it. – Hein