Detecting mouth with openCV

Question

I am trying to detect the mouth in an image with openCV, so I am using the following code:

#include "face_detection.h"

using namespace cv;

// Function detectAndDisplay
void detectAndDisplay(const std::string& file_name, cv::CascadeClassifier& face_cascade, cv::CascadeClassifier& mouth_cascade)
{
    Mat frame = imread(file_name);
    std::vector<Rect> faces;
    Mat frame_gray;
    Mat crop;
    Mat res;
    Mat gray;

    cvtColor(frame, frame_gray, COLOR_BGR2GRAY);
    equalizeHist(frame_gray, frame_gray);

    // Detect faces
    face_cascade.detectMultiScale(frame_gray, faces, 1.1, 3, 0 | CASCADE_SCALE_IMAGE, Size(30, 30));

    for(unsigned int i=0;i<faces.size();i++)
    {
        rectangle(frame,faces[i],Scalar(255,0,0),1,8,0);
        Mat face  = frame(faces[i]);
        cvtColor(face,face,CV_BGR2GRAY);
        std::vector <Rect> mouthi;
        mouth_cascade.detectMultiScale(face, mouthi);
        for(unsigned int k=0;k<mouthi.size();k++)
        {
        Point pt1(mouthi[k].x+faces[i].x , mouthi[k].y+faces[i].y);
        Point pt2(pt1.x+mouthi[k].width, pt1.y+mouthi[k].height);
        rectangle(frame, pt1,pt2,Scalar(0,255,0),1,8,0);
        }

    }

    imshow("Frame", frame);
    waitKey(33);
}

The classifiers are haarcascade_frontalface_alt.xml and haarcascade_mcs_mouth.xml.

The face is detected correctly but the mouth is not: I also obtain the eyes and some other parts, like the forehead. Is there a way to detect only the mouth?

Glomeration

Answer 1

I think I managed to solve the problem: focusing on the lower half of the face and increasing the scale factor did the trick and now I am able to detect the mouth with a good precision. Anyway this task seems much more complicated than face detection, even if I am using "simple" images, which means straight and full frontal.

Here are two examples: a success and a failure.

Glomeration

Hey, Can I get a link to the dataset from where you got these images? I have the images with me but I can't find the link to the page. I need to for citations. Thanks! – Proscenium

@AninditKarmakar Hi, it is the KDEF dataset: emotionlab.se/resources/kdef – Glomeration

@Glomeration Can you look on that topic: #37922413 – Siemens

Answer 2

I was facing the same problem, so I focused only on the lower half of the face and created an ROI from the detected face. It looks something like this:

Mat ROI=image(Rect(face.x,face.y+face.height*0.6,face.width,face.height*0.3));

Where face is the detected face from the image.

This created an ROI from the detected face for the lower half only. Else the mouth detector was detecting the eyes also as mouth.

Then use the MouthCascade.xml from this link: http://alereimondo.no-ip.org/OpenCV/34 which is far more efficient than the inbuilt OpenCV one.

Tisdale