Smooth representation change using Media Source API

Question

I am currently working on DASH player using JavaScript and MediaSource API

Streaming is working fine but I am stuck with changing representation. Probably the most bruteforced way to change representation during playback is about replacing <video> element in HTML document.

Hovewer, I was wondering if there is a simple way to implement adaptation (changing representation) with Media Source API. I've read that single Media Source object can handle many source buffers, but after adding second video buffer an exception is raised.

I am using Chrome 43.0.2357.65m

var mediaSource = MediaSource();
var url = URL.createObjectURL(mediaSource);

videoElement.src = url;

mediaSource.addEventListener('sourceopen', function () {
    var buffer1 = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.4d4015"');
    var buffer2 = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.4d4015"');
}

Exception:

Uncaught QuotaExceededError: Failed to execute 'addSourceBuffer' on    'MediaSource': This MediaSource has reached the limit of SourceBuffer objects it can handle. No additional SourceBuffer objects may be added.

Leatheroid

Answer 1

Unfortunately the MSE spec doesn't say much about smoothly changing representations. The way to learn how to do it is to read the code of an existing dash player and see how it does it. In order to save you a bunch of time, here is how to do it:

1. First of all you only need one source buffer. It doesn't matter that the codecs of the different representations are different. You create one source buffer and use it for the entire life span of the player. It doesn't matter how many times you switch representations.

2. The way to switch representation is actually very easy. You just have to append the initialization header of the target representation to the source buffer. And that's it. After that the decoder has been reinitialized and you can start appending segments that belong to the new representation.

3. Append segments of the target representation and enjoy the smooth transition.

Thant's it. It's not hard once you figure out what you have to do.

Chibouk

Svetlin Mladenov, regarding representation switching: as I understood from your answer, the buffer would look something like: INIT_q1, segment_q1, ..., INIT_q2, segment_q2, segment_q2, ... is this correct? Are there any requirements for the different representations of a video or would it work also if the target representation has different bandwidth + different resolution? – Bouzoun

@Silvia, yes that's correct. There are no limitations on the bandwidth or resolution. – Chibouk

I assume the primary thing that needs to be consistent is PTS of the frames, correct? – Woodcock

I was stuck an entire day with a problem when switch the video representation. You pointed me out, I just need to append the the initialization header whenever I switch. Thanks @SvetlinMladenov. – Clutter

Answer 2

I was also making an app to live stream from multiple cameras and faced the same issue when trying to play the stream of another camera after the player has already started playing from the feed of the first camera.

I was also trying to switch representation from a footage captured by one camera to another. I tried appending the initialization segment of the second camera before feeding the media source footage from the second camera so that it continues playing smoothly. But for some reason it didn't work.

So what I did instead was when capturing the videos I played them inside of a canvas element instead of a video element. When I change cameras, the video being played inside the canvas will change to the other camera and play smoothly.

Here is the trick, instead of streaming directly from the cameras, I was streaming from the canvas element! I used media recorder api to capture the contents of the canvas element and streamed that. Now the issue is solved, even if the cameras and representations appear to change, the actual representation of the footage being streamed didn't.

Deodorize

Hi there! This is an interesting solution. Can you post some code that shows how this works? – Draconic