Best way to record a HTML Canvas/WebGL animation server-side into a video?

Question

I have a set of animations which I can make in Canvas (fabric.js) or WebGL (three.js). I need to record them automatically, server-side, through a script and output a video file.

The animations include:

1. Pictures
   
2. Videos (with audio)
   
3. Other animations/effects

I have researched a lot during last few months on this.

Results
1. Use PhantomJS + FFMPEG
Run HTML Canvas animations on headless browser(PhantomJS) and record with FFMPEG. Here the issue is PhantomJS supports neither WebGL nor Video element. http://phantomjs.org/supported-web-standards.html

2. Use Websockets to send data back to server using DataURL
Here again, we will need to run the animations on browser (which we can't because we have to do everything on server).

3. Use node-canvas
This is a library by TJ Holowaychuk which allows rendering HTML Canvas on Node.js. But it has its own limitations plus I haven't really explored this field much. (If someone could shed more light on this library)

If anyone has done it before or can guide me somewhere useful.
All we need to do is use some data to create animations and record it into a video, everything on server side.

Yardmaster

node-canvas seems to be for 2d-contexts. I found the node-webgl package but have been unsuccessful to install its dependencies. However as a comment i have to write : if you want to record a webgl context (and use it as a rendering engine) to output videos for yourself, 1-by-1, you can do it server-side with xhr on a local server. But if you want to output videos to each connected user on the internet, just consider that it asks your serverSSS to have one CPU-GPU couple per visitor. I have been checking google for 'server-side 3D rendering' : there is nothing serious now with 2015 techs. – Fatigued

Also : what you describe is the way to use webgl as a rendering engine for yourself, that is why i wrote about it above. If you only look for the way to do it on a website your question is a duplicate of #64791. But of all the answers there, none brings anything ^^ – Fatigued

@Atrahasis What if we only want to record Canvas and NOT webGL. Can node-canvas be used for it? Does it support video and animations? – Yardmaster

ccapture.js: github.com/spite/ccapture.js – Pronto

cccapture does on the browser side right ..? – Cravat

@Pronto it needs animation to be run on the browser – Yardmaster

@Stallion yes, so we cant use it – Yardmaster

@Yardmaster Sure, you could use ccapture.js with any browser with xvfb. – Nolen

@ArtjomB. Can you elaborate a little? Or show some link or example? – Yardmaster

I haven't used xvfb myself, but I've seen success with selenium+Firefox here on SO. – Nolen

@abhinav Did you get the solution for this? Even i had the same problem statement wherein i had to record a webGL video. I am using plotly 3D Graph to be precise and tried all the options you mentioned above already. Also ccapture allows to stream the frames to a node server using ffmpegserver option. But it somehow freezes my UI while sending. If you have come up with any suitable solution, do let me know. – Decurved

Answer 1

You can use electron to render WebGL pages with BrowserWindow option "show" set to false and/or use xvfb-run to run headless.

Milliemillieme

Where does the rendering happen? Can I get a video for the scene that was just rendered saved on the filesystem? – Copperplate

Answer 2

I don't think node-canvas supports the webgl context, so you'll have to use a library built around 2d drawing, and it certainly won't have support for any video codecs.

If you can get your animation to work using node-canvas, you can grab the animated frames at a rate appropriate for your content, something like this:

Disclosure: I've successfully used FFmpeg to encode a sequence of externally generated images, but haven't tried the setInterval() method below. In addition to the animation overhead itself, I don't know how exporting a canvas to PNG files at 30 FPS would perform.

// assuming "canvas" is asynchronously drawn on some interval

function saveCanvas(canvas, destFile) {
  return new Promise((resolve, reject) => {
    const ext     = path.extname(destFile),
          encoder = '.png' === ext ?  'pngStream'
                                   : 'jpegStream';
    let writable = fs.createWriteStream(destFile),
        readable = canvas[encoder]();

    writable
      .on('finish', resolve)
      .on('error',  err => {
        let msg = `cannot write "${destFile}": ${err.message}`;
        reject(new Error(msg));
      });
    readable
      .on('end',   ()  => writable.end())
      .on('error', err => {
        let msg = `cannot encode "${destFile}": ${err.message}`;
        reject(new Error(msg));
      });
    readable.pipe(writable);
  });
}

const FPS = 30;

let frame    = 0,
    tasks    = [],
    interval = setInterval(() => tasks.push(
  saveCanvas(canvas, `frame_${frame++}.png`)), 1000 / FPS);

// when animation is done, stop timer
// and wait for images to be written
clearInterval(interval);

Promise.all(tasks).then(encodeVideo);

function encodeVideo() {
   // too much code to show here, but basically run FFmpeg
   // externally with "-i" option containing "frame_%d.png"
   // and "-r" = FPS. If you want to encode to VP9 + WEBM,
   // definitely see: http://wiki.webmproject.org/ffmpeg/vp9-encoding-guide
}

And then use FFmpeg to encode a sequence of images into a video.
For the code behind encodeVideo(), you can look at this example.

Edit: There may be an issue with canvas.pngStream() writing incorrect frames while the animation loop continuously draws on that one canvas--maybe a copy of the canvas needs to be created per frame? That would surely create significant memory pressure.

Handkerchief

Answer 3

I think that the chromium headless mode might support WebGL already and is another possibility. The video rendering part is yet to come though: https://bugs.chromium.org/p/chromium/issues/detail?id=781117

Ultann

Answer 4

CCapture.js makes this pretty easy.

Lexicostatistics