How can I write a frame of an embedded YouTube video (via iframe) to a canvas?

Question

I'd like to be able to extract frames from YouTube videos at various points within them (not just at the thumbnails), and do some processing on them. I can embed the video in my website using the iframe API, but I am struggling to find a way to capture that to a canvas. (It's ok if I am forced to capture the entire screen, and ok if I have to make changes to browser settings to allow it.)

Plethora

Along with the problem you should include the code you have up until this point (make some effort to answer your problem/question as Stack Overflow is not a code writing service). Please also include a specific problem or error and the shortest code necessary to reproduce it in the question itself. See: minimal reproducible example and also read How do I ask a Good Question. – Versicolor

Thanks. I'm not asking for code to be written, but I am asking for a general approach, as this problem is complicated by the security issue of iframes so it isn't all that straightforward. – Plethora

Answer 1

One option would be to write a browser extension. But I would guess you would want to avoid that.

The other option is to use the Screen Capture API (which is supported in Chrome, Edge and Firefox). Check out the browser compatibility information on MDN.

Using this API, you can allow your user to share his screen or browser tab and play this on a transient video element:

const videoElem = document.createElement('video');
videoElem.autoplay = true;

const displayMediaOptions = {
  video: {
    cursor: "never"
  },
  audio: false
};

async function startCapture() {
  videoElem.srcObject = await navigator.mediaDevices
    .getDisplayMedia(displayMediaOptions);
}

Later you can draw the video on a canvas to get a Base64 frame (basically a screenshot).

function getScreenshotInBase64() {
  let canvas = document.createElement('canvas');
  let context = canvas.getContext('2d');
  let [w, h] = [videoElem.videoWidth, videoElem.videoHeight];
  canvas.width =  w;
  canvas.height = h;
  context.drawImage(videoElem, 0, 0, w, h);
  return canvas.toDataURL();
}

Limitations with this approach:

• The user may select a different tab / application / screen or the page may be scrolled down and the video not visible.
• You get more than just the video (but based on your description, this is not necessarily bad).
• Pretty scarce browser support (if you consider older or mobile browsers).
• The user has to allow you to do it (basically each time you want to start sharing = recording the video in your case).

I build a small JsFiddle with the above snippets, which gets a Base64 print screen 2 seconds after you start screen sharing (by clicking the start button): https://jsfiddle.net/75Lmbf2o/.

Vivanvivarium

This should work. It says I can't award a bounty for 22 hours, I'll come back tomorrow and it is yours. You saved me a lot of time figuring that out, I appreciate it. :) – Plethora

Firefox 66 supports the Screen Capture API as well, and there's a polyfill available for older versions. – Fioritura

@jib: Thanks for the hint; I updated the answer accordingly. I somehow managed to misread the compatibility table initially :| – Vivanvivarium

@SerbanPetrescu You didn't. I updated it. Thanks for pointing it out! – Fioritura

Thanks again. Did exactly what I wanted! – Plethora

hey, how to do it with an browser extension?:) – Gherardi

Depends on the browser. For example, see developer.chrome.com/docs/extensions/reference/desktopCapture – Vivanvivarium

i added a full implementation for this approach: https://mcmap.net/q/1343300/-how-can-i-write-a-frame-of-an-embedded-youtube-video-via-iframe-to-a-canvas – Rapid