Template matching behavior on Color

Question

I am evaluating template matching algorithm to differentiate similar and dissimilar objects. What I found is confusing, I had an impression of template matching is a method which compares raw pixel intensity values. Hence when the pixel value varies I expected Template Matching to give a less match percentage.

I have a template and search image having same shape and size differing only in color(Images attached). When I did template matching surprisingly I am getting match percentage greater than 90%.

img = cv2.imread('./images/searchtest.png', cv2.IMREAD_COLOR)
template = cv2.imread('./images/template.png', cv2.IMREAD_COLOR)
res = cv2.matchTemplate(img, template, cv2.TM_CCORR_NORMED)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
print(max_val)

Template Image :

Search Image :

Can someone give me an insight why it is happening so? I have even tried this in HSV color space, Full BGR image, Full HSV image, Individual channels of B,G,R and Individual channels of H,S,V. In all the cases I am getting a good percentage.

Any help could be really appreciated.

Swaine

I am intrigued. Have you tried the same using a dark colored template? Something like dark blue/red? – Mackenzie

It is bothering me, since you asked I tried dark red template, it is matching for 75% though not getting the reason behind it – Swaine

From this we can conclude that it depends highly on the edges rather than color. Next, try matching a square (or any other shape) of the same color. – Mackenzie

I feel it will also match to some percent. Do you have any theory on its behaviour. What is it doing whether the raw pixel values itself or considers shape majorly – Swaine

It is likely that the matcher turns the images to grayscale before attempting to match. – Disentangle

@Yves: I have checked the OpenCV source code implementation and made sure there has been no conversion happening, yet not getting the physical interpretation and the math involved – Swaine

OpenCV actually uses different "Linear Spatial Filtering" methods instead of intensity comparison – Chiquita

From the formula being used in OpenCV(docs.opencv.org/2.4/modules/imgproc/doc/…), it seems they are using Intensity values to do the match. – Swaine

You need to do HS back projection before template matching – Teagan

Answer 1

res = cv2.matchTemplate(img, template, cv2.TM_CCORR_NORMED)

There are various argument, which you can use to find templates e.g. cv2.TM_CCOEFF, cv2.TM_CCOEFF_NORMED, cv2.TM_CCORR, cv2.TM_CCORR_NORMED, cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED

You can look into their equation here:

https://docs.opencv.org/2.4/modules/imgproc/doc/object_detection.html

From what I think if you want to use your template matching so that it doesn't match shape of different colours, then you should use CV_TM_SQDIFF or maybe cv2.TM_CCOEFF_NORMED. Correlation term gives matching for maximum value and Squared difference terms gives matching for minimum values. So in case you have exact shape and size, not same color though, you will get high value of correlation (see the equation in above link).

Arenas

Thanks, FYI I have tested all the 6 methods mentioned in the document and all of them matches the object even though of the color difference they have. – Swaine

Answer 2

Concept:

Suppose X=(X_1,X_2,....X_n), Y=(Y_1,Y_2,...,Y_n) satisfy Y_i=a * X_i for all i and some positive constant a, then (sum of all X_i * Y_i)=a * (Sum of (X_i)^2)=SquareRoot(Sum of (X_i)^2)*SquareRoot(Sum of (a * X_i)^2). therefore (sum of all X_i * Y_i)/(SquareRoot(Sum of (X_i)^2)*SquareRoot(Sum of (Y_i)^2))=1.

In your case, X represent your template image, almost only two color, background is black which is 0, the foreground color is constant c. Y represent ROI of your image, which is also almost only two color, background is 0, foreground color is another constant d. So we have a=d/c to satisfy above mentioned concept. So if we use cv2.TM_CCORR_NORMED, we get result near 1 is what we expected.

As for cv2.TM_CCOEFF_NORMED, if Y_i=a * X_i+b for all i and some constant b and some positive constant a, then correlation coefficient between X and Y is 1(Basic statistics). So if we use cv2.TM_CCOEFF_NORMED, we get result near 1 is what we expected.

Reynolds