Canvas 2D context really slow to draw hundreds or thousands of images

Question

I'm trying to create a multiplayer jigsaw puzzle game.

My first appoach was to use a <canvas> with a 2D rendering context, but the more I try, the more I think it's impossible without switching to WebGL.

Here is an example of what I got:

In this case I'm rendering a 1900x1200 pixel image cut into 228 pieces, but I want the game to be able to render thousands of pieces with higher resolution.

Each piece is procedurally generated using bezier curves & straight lines with random variations, which gives me these kind of result:

I need each pieces to be rendered independently as the players will be able to drag & drop them around to rebuild the puzzle.

Using clip()

At first, I only had one <canvas> and I used the clip() method, followed by a drawImage() call for each pieces.

But quickly ran into performance issues when trying to render hundreds of pieces at 60fps (I'm running this on an old laptop, but I feel like this is not the problem).

Here is a shortened version of the code I'm using:

class PuzzleGenerator {
  public static generatePieces(
    puzzleWidth: number,
    puzzleHeight: number,
    horizontalPieceCount: number,
    verticalPieceCount: number,
  ): Array<Piece> {
    const pieceWidth = puzzleWidth / horizontalPieceCount;
    const pieceHeight = puzzleHeight / verticalPieceCount;
    const pieces: Array<Piece> = [];
    for (let x = 0; x < horizontalPieceCount; x++) {
      for (let y = 0; y < verticalPieceCount; y++) {
        const pieceX = pieceWidth * x;
        const pieceY = pieceHeight * y;
        // For demonstration purpose I'm only drawing square pieces, but in reality it's much more complexe:
        // bezier curves, random variations, re-use of previous pieces to fit them together
        const piecePath = new Path2D();
        piecePath.moveTo(pieceX, pieceY);
        piecePath.lineTo(pieceX + pieceWidth, pieceY);
        piecePath.lineTo(pieceX + pieceWidth, pieceY + pieceHeight);
        piecePath.lineTo(pieceX, pieceY + pieceHeight);
        piecePath.closePath();
        pieces.push(new Piece(pieceX, pieceY, pieceWidth, pieceHeight, piecePath));
      }
    }
    return pieces;
  }
}

class Piece {
  constructor(
    public readonly x: number,
    public readonly y: number,
    public readonly width: number,
    public readonly height: number,
    public readonly path: Path2D,
  ) {}
}

class Puzzle {
  private readonly pieces: Array<Piece>;
  private readonly context: CanvasRenderingContext2D;

  constructor(
    private readonly canvas: HTMLCanvasElement,
    private readonly image: CanvasImageSource,
    private readonly puzzleWidth: number,
    private readonly puzzleHeight: number,
    private readonly horizontalPieceCount: number,
    private readonly verticalPieceCount: number,
  ) {
    this.canvas.width = puzzleWidth;
    this.canvas.height = puzzleHeight;
    this.context = canvas.getContext('2d') ?? ((): never => {throw new Error('Context identifier not supported');})();
    this.pieces = PuzzleGenerator.generatePieces(this.puzzleWidth, this.puzzleHeight, this.horizontalPieceCount, this.verticalPieceCount);
    this.loop();
  }

  private draw(): void {
    this.context.clearRect(0, 0, this.canvas.width, this.canvas.height);
    this.pieces.forEach((piece) => {
      this.context.save();
      this.context.clip(piece.path);
      this.context.drawImage(
        this.image,
        piece.x, piece.y, piece.width, piece.height,
        piece.x, piece.y, piece.width, piece.height,
      );
      this.context.restore();
      this.context.save();
      this.context.strokeStyle = '#fff';
      this.context.stroke(piece.path);
      this.context.restore();
    });
  }

  private loop(): void {
    requestAnimationFrame(() => {
      this.draw();
      this.loop();
    });
  }
}

const canvas = document.getElementById('puzzle') as HTMLCanvasElement;
const image = new Image();
image.src = '/assets/puzzle.jpg';
image.onload = (): void => {
  new Puzzle(canvas, image, 1000, 1000, 10, 10);
};

Using global composition

To try to improve performance, I switched from one to multiple <canvas> (one for each piece + one for the puzzle)

Each piece is drawn on its own offscreen canvas (MDN Optimizing Canvas) by filling its path and drawing the image on top using globalCompositeOperation = 'source-atop';

But it resulted in even worse performance. Even though each piece was drawn only one time in its own canvas, they were the same size as the entire puzzle, acting as layers, and each layer then had to be drawn into the puzzle's canvas each frame:

So I once again tried to optimize this, by reducing the canvas's size of each piece to the minimum, so they act more like sprites instead of layers (the spacing around each piece is to accommodate for the random variations):

Even though this optimization only removes transparent pixels, it has significantly increased the rendering performance.

At that point I'm able to draw hundreds of pieces at 60fps, but drawing thousands quickly drops me to 30fps or even lower.

To me, it looks like the 2D rendering context is having trouble to draw hundreds of images onto the same canvas, so whatever I do to improve the performance of drawing a single puzzle piece, it still won't be enough once I scale the puzzle to add more & more pieces and increase the resolution.

Other performance issues

Another problem I haven't addressed yet is that I want the players to be able to zoom in & out on the puzzle, but when I tried to zoom in my canvas using scale(), it also worsen the performance.

Also, I need to detect over which piece the player's mouse currently is. I'm using isPointInPath but I suspect it could become another performance issue in the long term.

Answers to questions you might ask me

Q: Why don't you draw the image only once and then draw the lines on top of it?
A: Yes, it is really fast to draw the puzzle that way, but this is not what I am looking for. My goal is to have a puzzle game where the pieces are scrambled at the beginning and the player can move them to rebuild the image.

Q: Have you tried building each piece only once on an offscreen canvas and then rebuild the entire puzzle at once on the visible canvas?
A: Yes I have, it helps, but there still is a linear performance decrease when I add more and more pieces to the puzzle.

Q: Why are you using an old laptop to debug this?
A: Well, the better the hardware, the better the framerate will be, but I'm building a jigsaw puzzle game, I feel like it should be able to run on low end hardware.

Q: What are your laptop specs?
A: It is a 7 year old laptop with an i7 4712HQ and a GT 750M (also it appears that the browser is using the integrated GPU instead of the dedicated GPU).

Q: Are you sure the performance issues come from the drawImage() and not from the bezier curves or other computations?
A: Yes, I'm sure, I simplified to the bare minimum and it appears that drawing a 50x50 pixel image a thousand times is slower than drawing a 100x100 pixel image 250 times (even though the global drawing resolution is the same at the end).

Demo

Here is a CodePen where you can play with the number of pieces to draw from the same source image.

It include a quick and dirty FPS counter to help you visualize performance drop.

drawImage() FPS profiling

And here is the TypeScript version without the FPS counter for better readability:

class Piece {
  constructor(
    public readonly x: number,
    public readonly y: number,
    public readonly width: number,
    public readonly height: number,
  ) {}
}

export class Puzzle {
  private readonly pieces: Array<Piece>;
  private readonly context: CanvasRenderingContext2D;
  private mousePosition?: {x: number; y: number};

  constructor(
    private readonly canvas: HTMLCanvasElement,
    private readonly image: CanvasImageSource,
    private readonly puzzleWidth: number,
    private readonly puzzleHeight: number,
    private readonly horizontalPieceCount: number,
    private readonly verticalPieceCount: number,
  ) {
    this.canvas.width = puzzleWidth;
    this.canvas.height = puzzleHeight;
    this.context = this.canvas.getContext('2d') ?? ((): never => {
      throw new Error('Context identifier not supported');
    })();
    this.pieces = this.generatePieces();
    this.trackMousePosition();
    this.loop();
  }

  private generatePieces(): Array<Piece> {
    const pieceWidth = this.puzzleWidth / this.horizontalPieceCount;
    const pieceHeight = this.puzzleHeight / this.verticalPieceCount;
    const pieces = [];
    for (let x = 0; x < this.horizontalPieceCount; x++) {
      for (let y = 0; y < this.verticalPieceCount; y++) {
        const pieceX = pieceWidth * x;
        const pieceY = pieceHeight * y;
        pieces.push(new Piece(pieceX, pieceY, pieceWidth, pieceHeight));
      }
    }
    return pieces;
  }

  private trackMousePosition(): void {
    this.canvas.addEventListener('mousemove', (event) => {
      this.mousePosition = {
        x: event.pageX - this.canvas.offsetLeft,
        y: event.pageY - this.canvas.offsetTop,
      };
    }, {passive: true});
  }

  private draw(): void {
    this.context.clearRect(0, 0, this.canvas.width, this.canvas.height);
    for (const piece of this.pieces) {
      this.context.drawImage(
        this.image,
        piece.x, piece.y, piece.width, piece.height,
        piece.x, piece.y, piece.width, piece.height,
      );
    }
    if (this.mousePosition) {
      this.context.beginPath();
      this.context.arc(this.mousePosition.x, this.mousePosition.y, 10, 0, 2 * Math.PI);
      this.context.fillStyle = '#f00';
      this.context.fill();
    }
  }

  private loop(): void {
    requestAnimationFrame(() => {
      this.draw();
      this.loop();
    });
  }
}

const canvas = document.getElementById('puzzle');
const image = new Image();
const imageWidth = 2000;
const imageHeight = 1000;
const horizontalPieceCount = 10;
const verticalPieceCount = 10;
image.src = `https://picsum.photos/${imageWidth}/${imageHeight}`;
image.onload = (): void => {
  new Puzzle(canvas, image, imageWidth, imageHeight, horizontalPieceCount, verticalPieceCount);
};

Can I solve this?

Is there some optimization I can try to improve the performance?

Am I doing something wrong that is killing the performance?

Have I reached the limit of what 2D rendering context can do?

Should I ditch 2D rendering context and switch to WebGL?

I was thinking about switching to PixiJS as it is a well know 2D rendering library, and it also has methods to draw shapes using bezier curves which may be useful to draw the puzzle's pieces.

Selfassured

Have you done any performance profiling to confirm that the slowdown is where you think it is? For example, maybe the performance issues are in the bezier curve logic and have little to do with drawing thousands of images. – Myology

I created a simple 2D tower defense games using nothing but standard browser libraries and a 2D context and I was able to render hundreds (maybe thousands) of individually rendered and animated sprites without any noticeable performance issues. github.com/cyborgx37/tower-defense-js – Myology

I guess the important part of rendering is to render as much as possible off-screen. The on-screen render should just be painting your off-screen canvas in a single step. See the render function here: github.com/cyborgx37/tower-defense-js/blob/main/src/js/… – Myology

What if you drew one big image, with a lot of lines / curves to show that it is split up into multiple pieces? – Juvenescent

Rather than creating a thousand canvases, have you considered creating a single puzzle piece canvas then drawing the puzzle pieces in a grid? Loading images into memory is notoriously expensive. The point of a sprite sheet is to allow the game engine to load a single image into memory, then draw parts from it. Most engines (including the 2D rendering context) are optimized to work this way. – Myology

Updated my answer to more clearly call out that you should not have thousands of canvases. – Myology

I haven't tried creating all the pieces in a grid on the same canvas, will definitely give it a try. Meanwhile, I created this codepen : codepen.io/julien-marcou/pen/zYRqdMB to demonstrate how I render the pieces, the demo runs at 30fps or lower on my old laptop, changing the piece count from 100x50 to 20x10 makes it run at a constant 60fps. – Selfassured

"on my old laptop"... Performance is complicated in that respect. If you are running this code on older architecture, it's just not going to be very fast, especially on a browser. The browser itself is already eating up a huge share of your system's resources, leaving you a preciously small margin to play with. If you want to support older machines, then you might have to make some sacrifices in the sizes of your puzzles, etc. – Myology

Yes, I understand, I feel like a jigsaw puzzle game should be able to run smoothly on this laptop, even though the puzzle has 5000 pieces. Using only one off-screen canvas to pre-render all the pieces is still not enought to make it run smoothly. Will have to try WebGL to see if it solves my problem. Meanwhile I update my original post with more info & a working demo the performance issue that I have. – Selfassured

Props on the demo. Excellent repro. On my machine there's no real frame rate hit until 60x60, which is a 3,600 piece puzzle, and even at 100x100, which is 10,000 pieces, the actual experience isn't terribly laggy for me (I would find it acceptable). However, after running some performance profiling on the code pen, I can confirm that the slow down is happening at the drawImage function call on the context. I think you are right that you are bumping into the limits of drawImage. I had some success with putImageData, but it's got problems too. – Myology

Actually... a bit of rounding seems to have solved the ugly grid problem... so maybe getImageData/putImageData is the trick. Just watch out for CORS issues with getImageData... it doesn't like "tainted" canvases. – Myology

I keep thinking about this issue and I remembered that in NES development it was common to render sprites in alternating groups. Technically, this would cause flickering, but then again that's how all screens work anyways. The idea, though, was that if you had a small enough group, the flickering is imperceptible, but it saves the GPU a lot of work. I experimented a bit and got much higher frame rates with higher piece counts, but it introduces other complications that would need to be accounted for. – Myology

Here's the Code Pen with the flicker technique: codepen.io/cyborgx37/pen/mdXEQXy. The opacity of the graph causes issues, but the image itself doesn't appear to flicker (to me). – Myology

Answer 1

Without performance profiling, it's anyone's guess where exactly the slowdown is happening. You are groping in the dark and making random changes hoping that it might have an effect.

That said, there are a couple common tips that are worth sharing. You will need to do the performance profiling first, but some of these might be helpful later.

Draw off-screen

Drawing to a visible canvas is a very expensive operation. You want to absolutely minimize the number of immediately visible draw operations. By comparison, drawing to a hidden canvas is very fast and efficient. Do all of your individual piece drawings to a hidden canvas, then draw that hidden canvas to the visible canvas in a single step (if possible).

E.g.:

const pieceLayer = document.createElement("canvas").getContext("2d");

// ...

for (const piece of this.pieces) {
  pieceLayer.draw(...);
}

// ...

visibleCanvas.draw(pieceLayer, ...);

Watch out for marginal performance hits in hot code paths

Despite what anyone may tell you, .forEach is not as fast as a good ol' for loop. .forEach has overheads that for doesn't. This is where profiling comes in, though, because it's hard to tell how much of a difference it would make for your code. Maybe none at all, maybe a noticeable amount. The differences are miniscule, but if you are processing very large collections many times a second, you'll start to see a difference. (Don't rely solely on advice: run tests for yourself which mimic your use-case as closely as possible.)

Pre-render expensive images

Pre-rendering is really critical. You can probably imagine that clip() is a relatively expensive operation. You are trying to use it a thousand times every 1/60th of a second.

You don't need to do that. Again, you will need to do some performance profiling to ensure that this is even worth the effort, but you can pre-render each piece and save it as an image instead of requiring the Canvas to constantly recalculate all of your clippings.

If you don't mind your game being static, then you can pre-render your images now, as part of the development process, and save them to a sprite sheet. It would look almost identical to the image you've already posted, where pieces are separated from each other in a grid. Then instead of expensive clipping you can just draw simple rectangles.

If you like creating dynamic puzzle pieces on each load (which does sound pretty cool) then you could still accomplish the same feat. As part of your initial load logic, use clip() to draw each puzzle piece to a single hidden sprite sheet canvas (in a regular grid pattern). Then in your actual render logic, you can take advantage of the pre-rendered pieces and greatly simplify the work the processor has to do.

Minimize the number of images/canvases

Loading images into memory so that they can be drawn to a canvas is a well-known performance hit. This is why sprite sheets are so common across all platforms... loading one large image and drawing sub-sections is faster than drawing hundreds of individual images which must be loaded then purged from memory.

Consider using getImageData/putImageData

This is for more extreme cases, but these two functions can save your canvas from doing a lot of work processing images. Essentially your can draw your image to a canvas, then use getImageData to extract the raw bitmap data which you would cache somewhere. Later, you can use putImageData to write the raw bitmap data to a canvas (maybe the same canvas).

However, these functions are complicated to use correctly (not least of all the CORS issues), so it's not a light 1:1 replacement.

Profile your code

These are some general tips that may or may not be effective for you. The most important advice though is what I started with: profile your code. You have to measure before you know where to cut. Working blind tends to make things worse, not better.

A personal example

Just for kicks, a while back I wrote a tower-defense game using no third-party libraries. I simply used a Canvas and a standard 2D Rendering Context. The game is definitely not finished, but it is able to render hundreds of individually animated sprites on a tiled background without any noticeable lag. The game supports dragging the viewport around in real time and works smoothly no matter what zoom level it's set to. You are welcome to look over the code for any tips or tricks that I've left out:

cyborgx37 / tower-defense-js

Myology

I'll update my original post with code closer to what I actually have, as it is not very clear, the code that I wrote in my post was the first approach that I had, but in fact, I'm a rendering each piece of the puzzle only once into their own off-screen canvas, and then rendering them all at once on the main visible canvas, which means the bezier curves computations as well as the clipping (or global composition) only occur once at the generation of the puzzle, then it's only drawing the off-screen canvas of all the pieces every frame – Selfassured

That would be helpful. – Myology

See, @RokoC.Buljan, this is why it's so important to do your own performance profiling rather than relying on advice you think you heard once in a talk. For most applications, there is no discernible difference between the two. You really have to be doing thousands of operations 60 times a second before you notice the difference. In those cases, it's all about the margins. Run the test for yourself and decide: gist.github.com/cyborgx37/ef60edea792e632f93b6e1992d744cdc – Myology

My results running Chrome 101.0.4951.54 (Official Build) (x86_64) on a MacBook Pro (15-inch, 2017) running MacOS 12.2.1 (21D62): Average Time For Each: 195.2020000000298, Average Time For Of: 156.7739999999851 – Myology