OpenCV/JavaCV face recognition - Very similar confidence values

Question

I will explain what I am trying to do, as it seems to be relevant in order to understand my question.

I am currently trying to do face recognition of people that step in front of a camera, based on known pictures in the database.

These known pictures are being collected from an identifying Smart Card (which contains only a single frontal face picture) or a frontal face profile picture from a social network. From what I've read so far, it seems that for a good face recognition, a good amount of training images is required (50+). As such, since my collected images are very few to create a reliable training set, I instead tried using my live camera frame captures (currently using 150) as the training set, and the identified pictures collected previously as the test set. I'm not sure if what I'm trying with this is correct, so please let me know if I'm screwing up.

So, the problem is that after I have let's say, 5 identified pictures that I got from Smart Cards, I tried to do face recognition using as a training set, the 150 frames which the camera captured of my face. When trying to recognize, the confidence values for each of the 5 test faces is EXTREMELY similar, making the whole program useless, because I cannot accurately recognize anyone. Often, using different camera captures as training I get higher confidence values from pictures of random people than the picture of myself.

I would appreciate any help you can give me, because I'm at a loss here.

Thank you.

Note: I'm using the JavaCV wrapper for OpenCV to make my program, and the haarcascades that come included in the package. Eigenfaces being the algorithm used.

Gossip

It's not clear which models you're training against what data...? – Featherveined

how so? if I understood your question correctly, I am using grayscaled, equalized, cropped face images in the training set. Using the same type of image as the test set also – Hygienic

Answer 1

Face Recognition

Intro

I want to add this. libfacerec has been included into the official OpenCV 2.4.2, see:

• http://opencv.org/opencv-v2-4-2-released.html

That means if you are using OpenCV 2.4.2, then you have the new cv::FaceRecognizer in the contrib module. I know a Python wrapper has been added lately (thanks for that!), probably Java is also wrapped at time of writing this.

cv::FaceRecognizer comes with an extensive documentation, that'll show you how to do face recognition with lots of full source code examples:

• http://docs.opencv.org/trunk/modules/contrib/doc/facerec/index.html

If you want to know how the available face recognition algorithms (Eigenfaces, Fisherfaces, Local Binary Patterns Histograms) work, then especially read the Guide To Face Recognition with OpenCV. In there I explain how the algorithms work and mention their shortcomings:

• http://docs.opencv.org/trunk/modules/contrib/doc/facerec/facerec_tutorial.html

Face Recognition with few images

Now to your original problem of recognizing faces, when your training dataset is small. I'll write you a thorough answer, so it probably helps people coming here from Google.

Actually Eigenfaces and Fisherfaces should not be used, when you only have very few samples per person in your data set. You need data for these models to work, I can't stress that enough. The more the better. These methods are based on estimating the variance in your data, so give them some data to estimate your model from! A while ago I ran a small test on the AT&T Facedatabase (with the facerec framework), which shows the performance of these methods with a varying number of images per person:

I am not writing a publication here, nor will I back these figures with a detailed mathematical analysis. It has been done before, so I recommend everyone doubting these figures to look into (2), in order to see a very detailed analysis of the PCA (Eigenfaces) and LDA (Fisherfaces) for small training data sets.

So what I suggest is using Local Binary Patterns Histograms (3) for Face Recognition in the small sample scenario. These are also included in the OpenCV FaceRecognizer and have been proven to perform very well on small training data sets. If you combine this with a TanTriggs Preprocessing (4), you should have a really robust Face Recognition model. The TanTriggs Preprocessing is a 8-liner (or so) in Python, see https://github.com/bytefish/facerec/blob/master/py/facerec/preprocessing.py#L41 for the implementation. That should be easy to adapt to Java (or I can implement it with OpenCV, if people request it).

Literature

• (1) Belhumeur, P. N., Hespanha, J., and Kriegman, D. Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 7 (1997), 711–720.
• (2) Martinez, A and Kak, A. PCA versus LDA IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No.2, pp. 228-233, 2001.
• (3) Ahonen, T., Hadid, A., and Pietikainen, M. Face Recognition with Local Binary Patterns. Computer Vision - ECCV 2004 (2004), 469–481.
• (4) Tan, X., and Triggs, B. Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Transactions on Image Processing 19 (2010), 1635–650.

Cylindrical

Thank you very much for your answer good sir, I have been getting a bit desperate with this, since I was not being able to get any decent results using my previous methods, even with the help of lukaskrieger which wrote the first answer (I still appreciate your help though!). I will have to try this out, and I will post here the results I am able to achieve. Thank you again for your insight, it is very much appreciated :) – Hygienic

Hello once more, I have been trying to find some sort of sample code/tutorial on how to this in java, since it seems it has been wrapped already - code.google.com/p/javacv/source/browse/src/main/java/com/…. However, I have not been successful in finding a tutorial, or doing it by myself... Some frustration later, I tried to give the c++ code a try, since it has the samples created, but I can't seem to get it to run, always returns: "‘imread’ was not declared in this scope" in function read_csv when compiling facerec_lbph.cpp. – Hygienic

Clearly the interface wrapped there isn't OpenCV 2.4.2. It may take some time for javacv to adapt. Local Binary Patterns Histograms are available in OpenCV and I've compiled the demo just yesterday for this question and it works without problems: https://mcmap.net/q/910002/-facerecognizer-in-opencv. Please make sure to read this answer, and if your problem persists you probably want to ask on answers.opencv.org/questions or file a bug. Bugs are not filed on stackoverflow.com, but in the bug tracker on code.opencv.org. – Cylindrical

Hello, my previous problem was fixed by upgrading my openCV from 2.4.1 to 2.4.2. I had assumed that any release after 2.4 would work. Ok, so now I can compile and test your example code facerec_lbph.cpp with my test csv file, and still got pretty bad results. I believe this happens because the images I used for the training were grayscale, histogram equalized images, and not with TanTriggs. I am not being able to apply this preprocessing method to my images, even after reading and trying out the python examples. Can you give any tips to point me in the right direction please? – Hygienic

Answer 2

What you want to know is how you can perform a face recognition with only one training image. This is possible but also depends on the number of different persons you want to classify.

50+ training images are certainly not needed. For a basic face recognition you need around 50 faces to calculate your face space (eigenfaces). Perhaps you mixed it up with that. See that you have lots of variations in this faces (skin colour, glasses, form ...) You can take these faces from any face database you like. http://www.face-rec.org/ lists several databases and explains different algorithms.

After you calculated your face space you train with as many faces you have. In your case you have only one. Depending on how many different subjects you want to classify this could already work.

If you get too many false classifications I would take a look at hybrid methods. Hybrid methods combine a template matching algorithm (eigenfaces, fisherfaces) with a feature based one. In this case you take the output of your first algorithm and match the eyes, nose, eyebrows, chin shape etc. with your test face.

In short:

1. extract faces from each image with haarcascades
2. calculate your face space
3. train for each face
4. ask for a face classification
5. take the most likely classifications and check for face features

In case you haven't found it, OpenCV has also a face recognition library: https://github.com/bytefish/libfacerec

EDIT: I wouldn't use more than 10-15 components (eigenfaces).

Weather

Hello, thanks for the reply :) I have also tried this way, using the Cambridge_FaceDB, where my training images are all the images in this face database PLUS my own identified pictures collected from the smart card. I am currently testing with inserting 3 collected pictures to the training set, and the test set contains 3 different pictures, which belong to those people. I am still getting confidence levels of around 95 to totally different people. ps: I am not currently using an hybrid method, but I even without this method I was expecting much different results. What else could cause this? – Hygienic

It still seems to me that you mix the face space generation with the training of your own faces. You don't need to add 3 collected faces to the training set. Only generate the face space with database images. This gives you a transformation to project faces into a coordinate system. Basically you apply this transformation to the faces of the subjects you want to identify later on (TRAINING). If you apply the transformation to a face you want to identify you project it into the same coordinate system and match with the nearest neighbour (MATCHING). Leave the confidence interval out for now. – Weather

Ok, I clearly am messing up somewhere in here.. So, I will only use one of the downloaded face databases as training, which will generate the average image and the eigenfaces correct? This next part is where I don't really understand. I am collecting images from the camera, and I want to see if this person in front of the camera is a known person (his face picture had been retrieved from his smart card). What I was doing before, was simply load the saved training data and then project the test image onto the PCA subspace in order to find the nearest neighbor, which always came out wrong. – Hygienic

Now you got it. In the subspace you need pre-classified faces. These are your training faces. When you project an unknown face into the space and find the nearest neighbour you get your classification. – Weather

This doesn't make sense to me, many face recognition guides using opencv and javacv say to create a text file which specifies where the training images are located (training images of people they want to recognize). If I am only using the Cambridge images as training, how can I expect to recognize myself in them? What it is currently doing is trying to match and determines a nearest neighbor that doesn't correspond to the test face... In case you're familiar with it, I've been following the sample code that comes in the javaCV package, "FaceRecognition.java". – Hygienic

Cambridge images for calculating the transformation to project into the subspace. Camera images for training. Then test. – Weather