Best metric for Face Embedding comparison during inference

Question

I want to run Face Recognition on CCTV footage. I have implemented several distance metrics for Face Embedding comparison during inference like Euclidean distance, Cosine distance, KDTree, SVM, L1 & L2 distance, etc but in the end I kept only the first two, as I was not getting expected accuracy from those it was difficult to find a good threshold.

def distance(self, embeddings1, embeddings2, distance_metric=0):
        if distance_metric == 0:
            # Euclidian distance
            embeddings1 = embeddings1/np.linalg.norm(embeddings1, axis=1, keepdims=True)
            embeddings2 = embeddings2/np.linalg.norm(embeddings2, axis=1, keepdims=True)
            dist = np.sqrt(np.sum(np.square(np.subtract(embeddings1, embeddings2))))
            return dist
        elif distance_metric == 1:
            # Distance based on cosine similarity
            dot = np.sum(np.multiply(embeddings1, embeddings2), axis=1)
            norm = np.linalg.norm(embeddings1, axis=1) * np.linalg.norm(embeddings2, axis=1)
            similarity = dot/norm
            dist = np.arccos(similarity) / math.pi
            return dist[0]
        else:
            raise 'Undefined distance metric %d' % distance_metric

Q1: What is the best metric in terms of accuracy for Face Embedding comparison and how to set a threshold in that?

Q2: What is the fastest and better way for Face Embedding comparison other than Euclidean and Cosine distance?

Hessite

Did you try to search Triplet-loss? – Checkered

I had trained my model using Triplet-loss earlier but here I want a distance metric for face comparison during recognition, not a Loss function. – Hessite

Answer 1

By best, if you mean state of the art - here's the list to follow: https://paperswithcode.com/sota/face-verification-on-labeled-faces-in-the

Currently, ArcFace is the best scoring model.

It uses Additive Angular Margin Loss for highly discriminative feature for face recognition.

Good thing is, it can be generalized easily and other loss functions can be designed based on the angular representation of features and weight-vectors including triplet loss.

For your question 2, it seems kind of vague. If you mean that how to make matchinf faster, you can always use clustering techniques instead of linear search.

The paper studies some acceleration studies too.

Code is available at:

https://github.com/deepinsight/insightface (mxnet)

https://github.com/TreB1eN/InsightFace_Pytorch (pytorch)

https://github.com/happynear/AMSoftmax (caffe)

For faster inference, you can use just L1 distance between the two embeddings, for practical purposes in inference stages simpler distance metrics are used. I would suggest you to experimentally observe which one works best in your trained model. Some other distance metrics https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics.pairwise

How to detect which face from the embedding database?

• The simplest approach is a linear scan. So, for all of the embeddings in your dataset, calculate the distance metric of your choice between the currently calculated face embedding and from the embedding database. Choose the one with minimum distance. Also, you may need to specify a threshold to discard unknown faces. Complexity O(N)

• A slightly better approach is to run an unsupervised clustering algorithm on your face embedding dataset to make multiple clusters. Let's say, there are k clusters, and in each cluster, there are on average p face embeddings.

You first search on all the clusters to find the one which is close to the current embedding O(k), then in that cluster, run a linear scan O(p) to find the suitable face.

• After that, it turns into a data structure problem. You can create a balanced tree to further speed up the matching.

Expressivity

Hi! I am using ArcFace for getting Face Embedding, but I looking for a better metric for embedding comparison after model inference, not a Loss Function or a model. Thanks! – Hessite

Better by which context, have you read the paper? I have not mentioned because of the loss function but because their angular distance metric. After model inference?? If you train your model with arcface do you think another distance metric will work better while you're doing inference? Your question is meaningless, kindly read the paper to get some idea. – Expressivity

Hi! Also, suppose you have trained an FR model using any loss function and you get an embedding vector out, then how will you use it to identify that person given you have thousands of know person's embedding in your DB? – Hessite

Can you please share some code? "calculate the distance metric of your choice" My question was only based on which is the best distance metric? – Hessite

As I clearly said, it's impossiblele to tell without any experimentation. I gave you the link to the SOTA paper, you can use L2 distance as they used but there are other options too. The 'best' needs some parameters (best in terms of accuracy (hard to tell, depends on dataset), fastest (L1 distance)). – Expressivity

Suppose you want to do Face Recognition on CCTV footages, then you don't have a fixed dataset, do you have any idea which metric will be better in terms of accuracy in that case? – Hessite

Answer 2

I recommend you to consume those state-of-the-art face recognition models within deepface. It offers a common interface and all you need is to pass model name as argument.

#!pip install deepface
from deepface import DeepFace
models = ['VGG-Face', 'Facenet', 'OpenFace', 'DeepFace'
           , 'DeepID', 'Dlib', 'ArcFace']
metrics = ['cosine', 'euclidean', 'euclidean_l2']
obj = DeepFace.verify("img1.jpg", "img2.jpg"
   , model_name = models[6], distance_metric = metrics[0])
print(obj)

Diplex

I tried deepface from Serengil but it is not able to verify many faces of same person and many times it returns true for faces of different person, do you know of any face comparison modules that are more accurate. What i tried, DeepFace.verify(img1_path="img1.png", img2_path="img2.png", model_name="Facenet", distance_metric="euclidean") – Champaigne

With its new version, deepface is able to verify many faces in an image. – Diplex