Measure of image similarity for feature matching?

Question

I'm currently trying to work with a Brute Force feature matcher using SIFT in openCV, using python. I'm trying to utilise it for my image search function on my server, where I'm inputting an image and having that image be compared with others, in the hopes that the matches will indicate a level of similarity. Is there a way to indicate a level of similarity via using feature matching?

Currently, I'm playing around with what I found on this website, which I'll also post below:

img1 = cv2.imread('box.png',0)          # queryImage
img2 = cv2.imread('box_in_scene.png',0) # trainImage

# Initiate SIFT detector
sift = cv2.SIFT()

# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)

# BFMatcher with default params
bf = cv2.BFMatcher()
matches = bf.knnMatch(des1,des2, k=2)

# Apply ratio test
good = []
for m,n in matches:
    if m.distance < 0.75*n.distance:
        good.append([m])

# cv2.drawMatchesKnn expects list of lists as matches.
img3 = cv2.drawMatchesKnn(img1,kp1,img2,kp2,good,flags=2)

plt.imshow(img3),plt.show()

What I'm using at the moment to create a measure of 'similarity' is the number of 'good' matches that are acquired from applying the ratio test, and just finding the how many 'good' matches are stored in good using a simple len(good).

This returned the number of good matches that I used to valuate the similarity of the input image to that of the database. However, I'm assuming it's not as simple as this, as when i began testing this using a picture of a shoe, images such as one of a banana, received a higher amount of 'good' matches than the other images of shoes. Even so far as to be more similar than the same shoe in a different colour.

I thought this may be just an anomaly, so I continued to test with a larger dataset of images, finding that again, the shoes weren't receiving scores (or number of good matches), as high as say an image of a quad-bike or a person, rather than matching with other shoes.

So basically, how can i define the similarity of two images using feature matching with a numerical value?

Thank you.

Autophyte

Answer 1

I think you need to choose better features in order to get better(or more similar images) results. SIFT is a local feature and there is a good chance you can find similar SIFT features even with images which are semantically different(as in shoe and banana).

To improve similarity accuracy, I would suggest you to decide on better features in addition to SIFT. Like colour histogram in the image. If you use colour histogram of the image, you will get images which are similar in colour histogram. You can use a good mix of features in order to find similarity. You can decide on this mix by checking what kind of images you have in database and what you feel could be discerning features between different semantic classes.

If you are fine to use a slightly different method, I would like to suggest PLSA, it is a method which I have used. Probabilistic latent semantic analysis(PLSA) is an unsupervised learning algorithm that represents data with a lower dimension hidden class. Similarity can be then found just by computing euclidean distance of the new images low dimensional representation with all the other classes. You can sort it based on the distance and get similar images. Even here choosing right features is important. Also you will need to choose number of hidden classes. You will need to experiment with number of classes.

I have one of my small projects which uses PLSA to solve image retrieval. So if you don't mind this plug, here it is PLSA Image retrieval. Unfortunately it is Matlab, but you can understand what is happening and try use it. I have used colour histogram as feature. So choose features which will help you discern different classes better.

Rance

Hey, thanks for the response! I've actually already done some testing using histograms and have even tried combining the histogram approach with feature matching, however, rather than supplementing the search results, the feature detection just seems to induce more false positives into the query result. I was hoping for the opposite effect as my final plan was to use a multitude of different methods and algorithms for searching and create a combination of sorts to provide the best possible matches, however from what I've tested with feature detection, it seems to have the opposite effect. – Autophyte

P.S. I've tried using SIFT, SURF and ORB with both BF matching and FLANN matching, however, I receive relatively similar results still. (My image index that I'm searching through is currently small, only searching through a set of roughly 20 to 50 images. – Autophyte

Cool! Thing is SIFT, SURF/ORB are all local features, even though they capture interesting features like corner points etc, they wont capture global structure of an object of interest. It is definitely not a straight forward problem. I would really bet on 1) unsupervised clustering (like PLSA or even just kmeans with SIFT). It is difficult to use global features like HOG here because the sample size is not big and the object classes are not fixed. second bet would be 2 ) use SIFT and colour histogram as bag of words model. Wish you all the best! – Rance

I'll give it a whirl! Thank you very much! – Autophyte

@Autophyte could you tell me which method gave you the best results ? – Heeler

I have good results with first aligning the two images using a feature detector (SIFT, ORB, etc.) and then comparing the aligned images using a correlation coefficient (Pearson, Spearman, etc.). I discuss this at <dsp.stackexchange.com/questions/52207/…>. However, this only works if the two images show the same thing exactly, it won't capture conceptual similarity (two pictures of different dogs, for example). I suppose you should try CNNs for that. – Unread