HTML5: Manually "playing" a video inside a canvas, in IOS Safari

Question

So, here's what I'm trying to do: I want to load a video into a video element, but not have it played in the "normal" way. Using a timed interval, calculated according to the movie's framerate, I want on each iteration to

A. Manually advance the video one 'frame' (or as close as possible to that).
B. Draw that frame into a canvas.

Thereby having the video "play" inside the canvas.

Here's some code:

<video width="400" height="224" id="mainVideo" src="urltovideo.mp4" type="video/mp4"></video>
<canvas width="400" height="224" id="videoCanvas"></canvas>

<script type="text/javascript">

var videoDom = document.querySelector("#mainVideo");
var videoCanvas = document.querySelector("#videoCanvas");
var videoCtx = null;
var interval = null;

videoDom.addEventListener('canplay',function() {
   // The video's framerate is 24fps, so we should move one frame each 1000/24=41.66 ms
   interval = setInterval(function() { doVideoCanvas(); }, 41.66);
});

videoDom.addEventListener('loadeddata',function() {
  videoCtx = videoCanvas.getContext('2d');
});

function doVideoCanvas() {
  videoCtx.drawImage(videoDom,0,0);
  //AFAIK and seen, currentTime is in seconds
  videoDom.currentTime += 0.0416;
}

</script>

This works perfectly in Google Chrome, but it doesn't work in an Iphone's Safari;
The video frames does not get drawn at all to the canvas.

I made sure that:

• The video events I hooked into does get triggered (did 'alerts' and they were shown).
• I have control over the canvas (did a 'fillRect' and it filled).

[I also tried specifying dimensions in the drawImage - it didn't help]
Is drawImage with a video object not applicable at all in Iphone Safari...?

Even if I'll manage to find a way to capture the video frames, there are also some other issues in the Iphone's browser:

• Access to the currentTime property of the video is only granted once the video has started playing (in a standard way). I thought about maybe somehow "playing it hidden" and then capturing, but didn't manage to do that. Also thought of maybe somehow start playing the video and then immediately stop it;
  There doesn't seem to be any way to forcefully start the playing of a video in the IOS Safari. video.play(), or autoplay doesn't seem to do anything. Only if the user taps the "play circle" on the video then the video starts playing (taking all over the screen, as usually with videos on the IPhone's browser).

• Once the video plays - the currentTime property does get forwarded. On the video itself. When you pause the video and go back to the normal html page - you can see the frames on the video element changing. Though, in a slow phase (unlike in Google Chrome, where the rate seems to be smooth enough to make it look like it's playing) - in the iphone it looks to be a rate of something like 2-3 frames per second maybe. It stays the same even if I try changing the interval timing, I guess there's a minimum time limit that the browser on the IPhone can handle.

"Bonus question" :) - When the frames on the video element progresses from the event - the circle "play button" is visible on the video element (since it is not actually 'playing'). Is there anyway to hide it and make it invisible?

This has been tested on Iphone 3GS (with both 3G and Wifi) and Iphone 4, both running IOS 5, both having the same results as described.

Empedocles

Answer 1

Unfortunately I don't have an iOS device to test this, but I don't think you need to actually capture the video frames in the way that you're attempting using the currentTime property. The usual process looks something like this:

1. create a video element without controls (omit the controls attribute)
2. hide the element
3. when the video is playing draw it to the canvas on an interval

As you've discovered, iOS devices do not support autoplay on HTML5 video (or indeed audio) but you can create a separate control to initiate playback of the video using the play() method.

This approach should solve the issue you're having with the play button being visible since in this case you are actually playing the video.

Conversazione

Thanks for the reply and the tips! I've encountered this document: developer.apple.com/library/safari/#documentation/AudioVideo/… Which has a line stating: "Note: Video as a source for the canvas drawImage() method is not currently supported on iOS." :\ But another option would be to convert the video to an image sequence - and use those on the canvas! – Empedocles

Good find Yuval, that's really useful to know. You should add this is an answer to your question and accept it. – Conversazione

similar capability seems to be lacking in Android Chrome and Android Browser at the moment too. – Banker

Answer 2

I don't believe the loadeddata event is called on iOS, try the loadedmetadata event instead. I also found it necessary on iOS to call the videoDom.load() method after setting videoDom.src.

For my use case, I need to do a "dRAF" (double requestAnimationFrame) after the seeked event to ensure something was actually drawn to the canvas rather than a transparent rectangle.

Try something like:

videoDom.onloadedmetadata = () => {
  videoCanvas.height = videoDom.videoHeight
  videoCanvas.width = videoDom.videoWidth
  videoDom.currentTime = 0
}

videoDom.onseeked = () => {
  // delay the drawImage call, otherwise we get an empty videoCanvas on iOS
  // see https://mcmap.net/q/1629085/-how-does-double-requestanimationframe-work
  window.requestAnimationFrame(() => {
    window.requestAnimationFrame(() => {
      videoCanvas.getContext('2d').drawImage(videoDom, 0, 0, videoCanvas.width, videoCanvas.height)
      videoDom.currentTime += 0.0416
    })
  })
}

videoDom.load()

From this gist

Darkroom

Will this work for realtime webcam stream using a VideoJS or something similar? – Kilovolt

I have no idea, try it. – Darkroom