HTML5 MediaSource works with some mp4 files and not with others (same codecs)
Asked Answered
S

3

8

I was playing around with MediaSource API. The code is taken directly from Mozilla's example page: https://developer.mozilla.org/en-US/docs/Web/API/MediaSource/endOfStream

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
  </head>
  <body>
    <video controls></video>
    <script>
      var video = document.querySelector('video');

      var assetURL = 'frag_bunny.mp4';
      // Need to be specific for Blink regarding codecs
      // ./mp4info frag_bunny.mp4 | grep Codec
      var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';

      if ('MediaSource' in window && MediaSource.isTypeSupported(mimeCodec)) {
        var mediaSource = new MediaSource;
        //console.log(mediaSource.readyState); // closed
        video.src = URL.createObjectURL(mediaSource);
        mediaSource.addEventListener('sourceopen', sourceOpen);
      } else {
        console.error('Unsupported MIME type or codec: ', mimeCodec);
      }

      function sourceOpen (_) {
        //console.log(this.readyState); // open
        var mediaSource = this;
        var sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
        fetchAB(assetURL, function (buf) {
          sourceBuffer.addEventListener('updateend', function (_) {
            mediaSource.endOfStream();
            video.play();
            //console.log(mediaSource.readyState); // ended
          });
          sourceBuffer.appendBuffer(buf);
        });
      };

      function fetchAB (url, cb) {
        console.log(url);
        var xhr = new XMLHttpRequest;
        xhr.open('get', url);
        xhr.responseType = 'arraybuffer';
        xhr.onload = function () {
          cb(xhr.response);
        };
        xhr.send();
      };
    </script>
  </body>
</html>

This works fine unless I use another video. I have a short test mp4 which - according to bento4's mp4info - is encoded with the same codecs as frag_bunny.mp4. It crashes at mediaSource.endOfStream(); because of

Uncaught DOMException: Failed to execute 'endOfStream' on 'MediaSource': The MediaSource's readyState is not 'open'.
    at SourceBuffer.<anonymous>

I assume that video file has to meet some specific criteria in order to work with that but I don't know them.

Can someone point me in the right direction?

Some more details about the files

frag_bunny mp4info

File:
  major brand:      mp42
  minor version:    1
  compatible brand: mp42
  compatible brand: avc1
  compatible brand: iso5

Movie:
  duration:   60095 ms
  time scale: 1000
  fragments:  yes

Found 4 Tracks
Track 1:
  flags:        7 ENABLED IN-MOVIE IN-PREVIEW
  id:           1
  type:         Audio
  duration: 60095 ms
  language: eng
  media:
    sample count: 0
    timescale:    22050
    duration:     0 (media timescale units)
    duration:     0 (ms)
    bitrate (computed): 65.075 Kbps
  Sample Description 0
    Coding:      mp4a (MPEG-4 Audio)
    Stream Type: Audio
    Object Type: MPEG-4 Audio
    Max Bitrate: 0
    Avg Bitrate: 64000
    Buffer Size: 6144
    Codecs String: mp4a.40.2
    MPEG-4 Audio Object Type: 2 (AAC Low Complexity)
    MPEG-4 Audio Decoder Config:
      Sampling Frequency: 22050
      Channels: 2
    Sample Rate: 22050
    Sample Size: 16
    Channels:    2
Track 2:
  flags:        7 ENABLED IN-MOVIE IN-PREVIEW
  id:           2
  type:         Video
  duration: 60095 ms
  language: eng
  media:
    sample count: 0
    timescale:    600
    duration:     0 (media timescale units)
    duration:     0 (ms)
    bitrate (computed): 612.178 Kbps
  display width:  640.000000
  display height: 360.000000
  Sample Description 0
    Coding:      avc1 (H.264)
    Width:       640
    Height:      360
    Depth:       24
    AVC Profile:          66 (Baseline)
    AVC Profile Compat:   e0
    AVC Level:            30
    AVC NALU Length Size: 4
    AVC SPS: [2742e01ea9181405ff2e00d418041adb0ad7bdf010]
    AVC PPS: [28de09c8]
    Codecs String: avc1.42E01E
Track 3:
  flags:        7 ENABLED IN-MOVIE IN-PREVIEW
  id:           3
  type:         Hint
  duration: 60095 ms
  language: eng
  media:
    sample count: 0
    timescale:    90000
    duration:     0 (media timescale units)
    duration:     0 (ms)
    bitrate (computed): 45.859 Kbps
  Sample Description 0
    Coding:      rtp  (RTP Hints)
Track 4:
  flags:        7 ENABLED IN-MOVIE IN-PREVIEW
  id:           4
  type:         Hint
  duration: 60095 ms
  language: eng
  media:
    sample count: 0
    timescale:    22050
    duration:     0 (media timescale units)
    duration:     0 (ms)
    bitrate (computed): 5.510 Kbps
  Sample Description 0
    Coding:      rtp  (RTP Hints)

my video's mp4info

File:
  major brand:      mp42
  minor version:    0
  compatible brand: mp42
  compatible brand: isom
  compatible brand: avc1

Movie:
  duration:   5568 ms
  time scale: 90000
  fragments:  no

Found 2 Tracks
Track 1:
  flags:        1 ENABLED
  id:           1
  type:         Video
  duration: 5533 ms
  language: und
  media:
    sample count: 166
    timescale:    90000
    duration:     498000 (media timescale units)
    duration:     5533 (ms)
    bitrate (computed): 465.670 Kbps
  display width:  560.000000
  display height: 320.000000
  frame rate (computed): 30.000
  Sample Description 0
    Coding:      avc1 (H.264)
    Width:       560
    Height:      320
    Depth:       24
    AVC Profile:          66 (Baseline)
    AVC Profile Compat:   c0
    AVC Level:            30
    AVC NALU Length Size: 4
    AVC SPS: [6742c01e9e218118534d40404050000003001000000303c8f162ee]
    AVC PPS: [68ce06cb20]
    Codecs String: avc1.42C01E
Track 2:
  flags:        3 ENABLED IN-MOVIE
  id:           2
  type:         Audio
  duration: 5568 ms
  language: eng
  media:
    sample count: 261
    timescale:    48000
    duration:     267264 (media timescale units)
    duration:     5568 (ms)
    bitrate (computed): 83.050 Kbps
  Sample Description 0
    Coding:      mp4a (MPEG-4 Audio)
    Stream Type: Audio
    Object Type: MPEG-4 Audio
    Max Bitrate: 91632
    Avg Bitrate: 83051
    Buffer Size: 280
    Codecs String: mp4a.40.2
    MPEG-4 Audio Object Type: 2 (AAC Low Complexity)
    MPEG-4 Audio Decoder Config:
      Sampling Frequency: 48000
      Channels: 1
    Sample Rate: 48000
    Sample Size: 16
    Channels:    1

Chrome version: 55.0.2883.95 (64-bit)

Shopworn answered 14/2, 2017 at 18:55 Comment(0)
S
6

t works fine if mp4 is fragmented. You can do that using bento4's mp4fragment tool which you can find here: https://www.bento4.com/documentation/mp4fragment/

Shopworn answered 15/2, 2017 at 5:6 Comment(0)
H
4

To build on Xander's answer, it is useful to look at the way the video is packaged and streamed:

  • camera camptures 'raw' image - generally a large bitmap of pixels
  • encoders encode each frame into a more efficient format for storage and transporting - example encoders are h.264, h.265, vp9 etc. It is worth nothing there are variants of each and different profiles within some also (so one h.264 encoded video may play on a particular device/client and another may not).
  • the video stream is packaged into a container, e.g. mp4. This container will typically contain streams for video, audio, metadata, subtitles (if present) etc.
  • If the video is to be streamed with an Adaptive Bit Rate streaming protocol such as HLS or DASH, the video container will be broken up into segments of equal length, e.g. 10 second segments, which the client can request one at a time. If the video is made available in multiple bit rate video streams, this allows the client select the next segment from the bit rate that best matches the current network conditions.

As Xander's says, the code above is designed to handle an MP4 file that has been segmented or fragmented in this way, and not a non-fragmented mp4 video file.

Hight answered 16/2, 2017 at 12:29 Comment(0)
O
3

Let me add that in Chrome it is not enough to have a segmented MP4, a segmented MP4 may still produce the error that you got.

So in chrome you have to follow the advice from MDN - in a nutshell if you are using ffmpeg, you should format your MP4 with this command:

ffmpeg -i input.mp4 -movflags frag_keyframe+empty_moov+default_base_moof output.mp4
Obstacle answered 20/1, 2023 at 21:34 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.