Matching image and determine best match using SURF

Question

I have been trying to use the EMGU Example SURFFeature to determine if an image is in a collection of images. But I am having problems understanding how to determine if a match was found.

.........Original image ..............................Scene_1 (match).........................Scene_2 (no match)

................... ...................

I have been looking at the documentation and spent hours looking for a possible solution, on how to determine if the images are the same. As you can see in the following pics, a match is found for both.

Its clear that the one I'm trying to find gets more matches (lines connecting) but how do I check this in the code?

Question: How do I filter out the good match?

My goal is to be able to compare an input Image (capture from webcam) with a collection of images in a database. but before I can save all images to the DB I need to know what Values I can compare the input to. (e.g. save the objectKeypoints in the DB)

Here is my sample code (the matching part):

private void match_test()
{
    long matchTime;
    using (Mat modelImage = CvInvoke.Imread(@"images\input.jpg", LoadImageType.Grayscale))
    using (Mat observedImage = CvInvoke.Imread(@"images\2.jpg", LoadImageType.Grayscale))
    {
        Mat result = DrawMatches.Draw(modelImage, observedImage, out matchTime);
        //ImageViewer.Show(result, String.Format("Matched using {0} in {1} milliseconds", CudaInvoke.HasCuda ? "GPU" : "CPU", matchTime));
        ib_output.Image = result;
        label7.Text = String.Format("Matched using {0} in {1} milliseconds", CudaInvoke.HasCuda ? "GPU" : "CPU", matchTime);
     }
}

public static void FindMatch(Mat modelImage, Mat observedImage, out long matchTime, out VectorOfKeyPoint modelKeyPoints, out VectorOfKeyPoint observedKeyPoints, VectorOfVectorOfDMatch matches, out Mat mask, out Mat homography)
{
    int k = 2;
    double uniquenessThreshold = 0.9;
    double hessianThresh = 800;

    Stopwatch watch;
    homography = null;

    modelKeyPoints = new VectorOfKeyPoint();
    observedKeyPoints = new VectorOfKeyPoint();

    using (UMat uModelImage = modelImage.ToUMat(AccessType.Read))
    using (UMat uObservedImage = observedImage.ToUMat(AccessType.Read))
    {
        SURF surfCPU = new SURF(hessianThresh);
        //extract features from the object image
        UMat modelDescriptors = new UMat();
        surfCPU.DetectAndCompute(uModelImage, null, modelKeyPoints, modelDescriptors, false);

        watch = Stopwatch.StartNew();

        // extract features from the observed image
        UMat observedDescriptors = new UMat();
        surfCPU.DetectAndCompute(uObservedImage, null, observedKeyPoints, observedDescriptors, false);

        //Match the two SURF descriptors
        BFMatcher matcher = new BFMatcher(DistanceType.L2);
        matcher.Add(modelDescriptors);

        matcher.KnnMatch(observedDescriptors, matches, k, null);

        mask = new Mat(matches.Size, 1, DepthType.Cv8U, 1);
        mask.SetTo(new MCvScalar(255));

        Features2DToolbox.VoteForUniqueness(matches, uniquenessThreshold, mask);
        int nonZeroCount = CvInvoke.CountNonZero(mask);

        if (nonZeroCount >= 4)
        {
            nonZeroCount = Features2DToolbox.VoteForSizeAndOrientation(modelKeyPoints, observedKeyPoints,
               matches, mask, 1.5, 20);

            if (nonZeroCount >= 4)
                homography = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(modelKeyPoints,
                   observedKeyPoints, matches, mask, 2);
        }

        watch.Stop();
    }

    matchTime = watch.ElapsedMilliseconds;
}

I really have the feeling I'm not far from the solution.. hope someone can help me out

Synchrocyclotron

if you can take better input pictures, this approach should probably work well. – Demagogy

The approach works sort of.. But my question still stands. I am looking for a way to measure if there was a good match or if a match was bad. By eye it seems there must be an obvious solution i am missing. – Synchrocyclotron

Check this. It tells you if the homography is good, i.e. if you have a good match. Could be a good starting point. Or you can wrap one image on the other, and check for absolute difference of pixel intensities, or the like. – Demagogy

Thanks this did help. and after 5 checks it seems to work nicely. But beware: I am using Emgu 3.0 (This link helped me with the port. Also I had to copy my homography Mat to a new Image<Gray,Byte> or it would crash. – Synchrocyclotron

Ok good ;D I'm not familiar with C# wrappers – Demagogy

unfortunately I still get a lot of false matches.. – Synchrocyclotron

You could try comparing the coordinates of the matches (match 1 on img1 with match 1 on img2) and see how closely they lie over each other to get more information on the quality of the matches. – Polycythemia

maybe try what's suggested in the following post: #24569886 – Polycythemia

Answer 1

On exit from Features2DToolbox.GetHomographyMatrixFromMatchedFeatures, the mask matrix is updated to have zeros where matches are outliers (i.e., don't correspond well under the computed homography). Therefore, calling CountNonZero again on mask should give an indication of match quality.

I see you're wanting to classify matches as "good" or "bad" rather than just compare multiple matches against a single image; from the examples in your question it looks like maybe a reasonable threshold would be 1/4 the keypoints found in the input image. You might want an absolute minimum as well, on the grounds that you can't really consider something a good match without a certain quantity of evidence. So, e.g., something like

bool FindMatch(...) {
    bool goodMatch = false;
    // ...
    homography = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(...);
    int nInliers = CvInvoke.CountNonZero(mask);
    goodMatch = nInliers >= 10 && nInliers >= observedKeyPoints.size()/4;
    // ...
    return goodMatch;
}

where on branches that don't get as far as computing homography of course goodMatch just stays false as it was initialized. The numbers 10 and 1/4 are kinda arbitrary and will depend on your application.

(Warning: the above is entirely derived from reading the docs; I haven't actually tried it.)

Mesenchyme

I see I got a downvote for this; if whoever downvoted it happens to be reading this comment, I'd be interested to know what you found unsatisfactory so that I can improve it if possible. – Mesenchyme

Well I didn't. Thanks alot for your answer! – Synchrocyclotron

You're welcome. I hope it does turn out to be helpful. – Mesenchyme