Is there any way to auto crop the face after doing face detection with face-api.js?

Question

I've implemented face-API in my react project which is detecting a single face with detectSingleFace from the picture.

Now I want to move one step further. I want face-api to auto-crop the face after detection. So, I can store it in some server, state or local storage. Is there any way to do so?

Here you can see a screenshot example I want to achieve One side is a picture another side is the auto cropped face(which I want to implement).

Here is my live code link in codesandbox

Below is my code module for face-api

PhotoFaceDetection.js

import React, { useState, useEffect, useRef } from "react";
import * as faceapi from "face-api.js";
import Img from "./assets/mFace.jpg";
import "./styles.css";

const PhotoFaceDetection = () => {
  const [initializing, setInitializing] = useState(false);
  const [image, setImage] = useState(Img);
  const canvasRef = useRef();
  const imageRef = useRef();

  // I want to store cropped image in this state
  const [pic, setPic] = useState();

  useEffect(() => {
    const loadModels = async () => {
      setInitializing(true);
      Promise.all([
        // models getting from public/model directory
        faceapi.nets.tinyFaceDetector.load("/models"),
        faceapi.nets.faceLandmark68Net.load("/models"),
        faceapi.nets.faceRecognitionNet.load("/models"),
        faceapi.nets.faceExpressionNet.load("/models")
      ])
        .then(console.log("success", "/models"))
        .then(handleImageClick)
        .catch((e) => console.error(e));
    };
    loadModels();
  }, []);

  const handleImageClick = async () => {
    if (initializing) {
      setInitializing(false);
    }
    canvasRef.current.innerHTML = faceapi.createCanvasFromMedia(
      imageRef.current
    );
    const displaySize = {
      width: 500,
      height: 350
    };
    faceapi.matchDimensions(canvasRef.current, displaySize);
    const detections = await faceapi.detectSingleFace(
      imageRef.current,
      new faceapi.TinyFaceDetectorOptions()
    );
    const resizeDetections = faceapi.resizeResults(detections, displaySize);
    canvasRef.current
      .getContext("2d")
      .clearRect(0, 0, displaySize.width, displaySize.height);
    faceapi.draw.drawDetections(canvasRef.current, resizeDetections);
    console.log(
      `Width ${detections.box._width} and Height ${detections.box._height}`
    );
    setPic(detections);
    console.log(detections);
  };

  return (
    <div className="App">
      <span>{initializing ? "Initializing" : "Ready"}</span>
      <div className="display-flex justify-content-center">
        <img ref={imageRef} src={image} alt="face" crossorigin="anonymous" />
        <canvas ref={canvasRef} className="position-absolute" />
      </div>
    </div>
  );
};

export default PhotoFaceDetection;

Peoria

Just looked at your sandbox and it seems like you've already got it working, is that correct? You could post that as an answer. – Frederique

@BasvanderLinden sure – Peoria

Answer 1

After doing a lot of R&D I figured it out. For future readers who may face an issue here is the guide. I've created another function that will get the original image reference and the bounded box dimension i.e. width and height. After that, I've used faceapi method to extract faces and then with the help of the toDataURL method I actually converted it to base64 file which can be rendered to any image src or can be stored anywhere. This is the function I was explaining above

async function extractFaceFromBox(imageRef, box) {
    const regionsToExtract = [
      new faceapi.Rect(box.x, box.y, box.width, box.height)
    ];
    let faceImages = await faceapi.extractFaces(imageRef, regionsToExtract);

    if (faceImages.length === 0) {
      console.log("No face found");
    } else {
      const outputImage = "";
      faceImages.forEach((cnv) => {
        outputImage.src = cnv.toDataURL();
        setPic(cnv.toDataURL());
      });
      // setPic(faceImages.toDataUrl);
      console.log("face found ");
      console.log(pic);
    }
  }

Then I call the above function inside my main function where I used faceapi face detection tiny model. extractFaceFromBox(imageRef.current, detections.box);

You can also visit live code here to check complete implementation

Peoria