Get frame numbers in HTML5 Video

Question

I am trying to capture each frame number of the video however it looks like there is no way of achieving it. So I started my own clock to match the frame numbers of the video but they never match and the difference keeps increasing as the video progress.

Please have a look at my bin. http://jsbin.com/dopuvo/4/edit

I have added the frame number to each frame of the video from Adobe After Effect so I have more accurate information of the difference. The Video is running at 29.97fps and the requestAnimationFrame is also set to increase the number at the same rate, however I am not sure where this difference is coming from.

Sometimes they match and sometimes they don't. I also tried doing it offline but I get the same results. Any help.

Vena

Thanks Samuel, and yes, infact this is the solution I followed but I need to keep getting the frame numbers as the video moves forward. Also I tried without rounding off the total frames but no luck. – Vena

It might be possible (if you control the canvas it is drawing on) to keep a counter there: kaizou.org/2012/09/frame-by-frame-video-effects-using-html5-and though that also looks like dark magic from my perspective LOL see also #17045067 – Protasis

Answer 1

I found something on github for this. https://github.com/allensarkisyan/VideoFrame

I have implemented it in this fiddle: https://jsfiddle.net/k0y8tp2v/

var currentFrame = $('#currentFrame');
var video = VideoFrame({
    id : 'video',
    frameRate: 25,
    callback : function(frame) {
        currentFrame.html(frame);
    }
});

$('#play-pause').click(function(){
    if(video.video.paused){
        video.video.play();
        video.listen('frame');
        $(this).html('Pause');
    }else{
        video.video.pause();
        video.stopListen();
        $(this).html('Play');
    }
});

EDIT: updated fiddle to new video so it works again.

EDIT: As pointed out, the video is 25fps, so I updated it, and while I was there removed reliance on jQuery.
Non jQuery version:
https://jsfiddle.net/k0y8tp2v/1/

var currentFrame = document.getElementById('currentFrame');
var video = VideoFrame({
    id : 'video',
    frameRate: 25,
    callback : function(frame) {
        currentFrame.innerHTML = frame ;
    }
});

document.getElementById('play-pause').addEventListener('click', function(e){
    if(video.video.paused){
        video.video.play();
        video.listen('frame');
        e.target.innerHTML = 'Pause';
    }else{
        video.video.pause();
        video.stopListen();
        e.target.innerHTML = 'Play';
    }
});

Hersey

Sometimes it is 1 frame off when the video is stopped, but this could probably be fixed by setting frame number html again when the video is paused/stopped – Hersey

Appears that that library accomplishes the framerate tracking by setting a timer that polls it "twice per expected framerate" for followers :) – Protasis

frameRate should match the actual one from the video. Setting it to 100 does not affect the playback speed, however it does affect the frame counting. – Insensible

Thanks for your code. I checked it using ffmpeg software and quick time player. Sometimes count of frames given by videoframe is 1-2 different from count by ffmpeg. Is there any another way to get accourate frame number in javascript? – Keirakeiser

@Hersey Thanks for the fiddle. The frameRate for the current video (Big Buck Bunny) is 25, not 29.97. When you correct that it is giving a more accurate frame number. – Sha

@Dipu Raj, ahh, you are correct, I updated the fiddle and code. Wow, answer 2 days short of 5 years old and still helping people. It gives me a warm fuzzy feeling :) – Hersey

Answer 2

The problem is that setTimeout is not really predictable, so you can't be sure that exactly one new frame has been displayed every time your function runs. You need to check the currentTime of the video every time you update your frame display and multiply that by the frame rate.

Here's a working example: http://jsbin.com/xarekice/1/edit It's off by one frame, but it looks like you may have two frames at the beginning marked "000000".

A few things about the video element that you may want to be aware of:

1. As you seem to have discovered, there's no reliable way to determine the frame rate, so you have to discover it elsewhere and hard-code it. There are some interesting things going on with video metrics, but they're non-standard, not widely supported and, in my experience, completely ineffective at determining the actual frame rate.

2. The currentTime is not always exactly representative of what's on the screen. Sometimes it's ahead a little bit, especially when seeking (which is why in my JSBin, I don't update the frame while you're seeking).

I believe currentTime updates on a separate thread from the actual video draw, so it kind of works like it's own clock that just keeps going. It's where the video wants to be, not necessarily where it is. So you can get close, but you need to round the results of the frame calculation, and once in a while, you may be off by one frame.

Banneret

Answer 3

Starting in M83, Chromium has a requestVideoFrameCallback() API, which might solve your issue. You can use the mediaTime to get a consistent timestamp, as outlined in this Github issue. From there, you could try something like this:

var frameCounter = (time, metadata) => {
   let count = metadata.mediaTime * frameRate;
   console.log("Got frame: " + Math.round(count));

   // Capture code here.

   video.requestVideoFrameCallback(frameCounter);
}

video.requestVideoFrameCallback(frameCounter)

This will only fire on new frames, but you may occasionally miss one (which you can detect from a discontinuity in the metadata.presentedFrames count). You might also be slightly late in capturing the frame (usually 16ms, or one call to window.requestAnimationFrame() later than when the video frame is available).

If you're interested in a high level overview of the API, here's a blogpost, or you can take a look at the API's offical github.

Pelerine