Recognising objects in images using HAAR cascade and OpenCV

Question

(This came out a bit long, but it's mostly an hardly hard to understand explanation :)

For a project I have I need to recognise objects that, in general form, look like this -

Inside a bigger image that contains different shapes, like this one -

As you can see, the object I'm looking for is a red line with crosses on each side (there's 5 of them in that last picture). I have a bank of around 4,000 images in which I need to find the object, some of them contain these objects, and some of them don't, like this image -

After doing some research, I've figured using haar cascades and openCV is the way to go, so I wrote a script that goes through all of the aforementioned 4,000 images and extracts separates contours, like the first image in this question.

Then, I went through the many contours, grabbed around 150 of them (that is, 150 files that contain just the object I need, similar to the first image) and around 180 images that do not contain the object I need (similar to the third picture here).

Then I started the training process, using several tutorials, but mainly this one.

While doing so, I encountered a problem - as you can see, the images of the desired double-crossed object are not the same size, and don't even have the same scale (as they can appear in any angle - horizontally, diagonally, etc..).

At first I tried using the images with the different dimension, but that caused errors in the training process, so, to work around that, I've changed all of the positive images' dimension to be 350x350 (the biggest scale of one of the objects). Just to be clear - I did not resize the images - I just added white space around to make all of the images to be 350x350 pixels.

Then I went through the training process, as suggested in the tutorial - I created samples (width - 24, height - 24) and created a cascade xml file, which turned out to be not very big (45kb).

Now, I know that 150 positive images and 180 negative ones are not a lot, but I wanted to at least get a proof-of-concept working before I filtered more images and put more time into it.

When the cascade.xml file was done, I tried to use it to locate some objects in some images (using cv2.CascadeClassifier('cascade.xml') and cascade.detectMultiScale(img) but every try returned zero results.

At last I even tried to locate an object in one of the positive images (which contained nothing but one of the desired objects), but it too returned zero results.

I tried tweaking the parameters of cascade.detectMultiScale(img) and currently I'm training a cascade file with 36x36 samples, but I'm not confident it will work.

Since I'm pretty new to this stuff, I was wondering what I'm doing wrong, and I thought I'll ask here.

More specifically:

• Do you think the use of haar is right in this context? Should I use other method of objects recognition?
• Could the positive images dimensions be the source of problem? If so, how can I go about it?

If I didn't include some important data, please let me know I'll post it.

Thank you very much for your help, Dan

Fertile

Answer 1

i guess, you won't get good results from haar (or hog) cascade classifiers here.

• your 'needle' does not have enough features/corners (it's just 2 crosses and a line)
• cascade classifiers are quite sensitive to rotation. it seems your object can take any arbitrary rotation here.
• if you train a classifier with many different rotations, it will just overfit.
• if you train many classifiers(one per rotation), - the same. ;(

so, imho, not much hope for that approach.

i would go for contours/shapeMatching instead:

void findNeedles( const std::vector<cv::Point> & needle_contour, const cv::Mat & haystack_binarized)
{
    int nfound = 0;
    std::vector<std::vector<cv::Point>> contours;
    cv::findContours(haystack_binarized, contours, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_NONE);
    for (size_t i = 0; i < contours.size(); i++)
    {
        // pre-filter for size:
        if ( ( contours[i].size() < needle_contour.size()/2 )
          || ( contours[i].size() > needle_contour.size()*2 ) )
          continue;

        double d = cv::matchShapes(contours[i],needle_contour,CV_CONTOURS_MATCH_I2,0);
        if ( d < 8.4 ) // heuristic value, experiments needed !!
        {
            cv::drawContours(haystack_binarized, contours, i, 128, 3);
            nfound ++;
        }
    }
    cerr << nfound << " objects found" << endl;
    cv::imshow("haystack",haystack_binarized);
    //imwrite("hay.png",haystack_binarized);
    cv::waitKey();
}


int main()
{
    // 1. get the contour of our needle:
    Mat needle = imread("needle.png",0);
    Mat needle_b; 
    threshold(needle,needle_b,120,255,1); 
    imshow("needle",needle_b);

    std::vector<std::vector<cv::Point>> needle_conts;
    cv::findContours(needle_b, needle_conts, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_NONE);
    if ( needle_conts.size() == 0 )
    {
        std::cout << " no contour Found" << std::endl;
        return -1;
    }
    std::vector<cv::Point> needle_contour = needle_conts[0];

    // 2. check a positive sample:
    Mat haypos = imread("hay_pos.png",0);
    Mat haypos_b; 
    threshold(haypos,haypos_b,120,255,1);
    findNeedles(needle_contour, haypos_b);

    // 3. check a negative sample:
    Mat hayneg = imread("hay_neg.png",0);
    Mat hayneg_b; 
    threshold(hayneg,hayneg_b,120,255,1);
    findNeedles(needle_contour, hayneg_b);

    return 0;
}

--------------

> haystack.exe
5 objects found
0 objects found

Claritaclarity

I agree, haar it is not you would like to use here. – Aurilia

First of all - thank you very much. You solution was very helpful and explained exactly what I needed to do. I also appreciate very much the time you took to write the code. Because of people like you SO is as great as it is. So again - thank you. Secondly, for general knowledge - the solution recognised some long lines that passed the size filter as needles, so to go about it I matched a given contour to the needle and to another contour of a line, and compared the grades to deduce which kind of object it is. – Fertile