Template matching with multiple objects in OpenCV Python

Question

I'm trying to find multiple templates in an image using opencv python, according to this link.

But the problem is that multiple points returned for a single object with a slightly difference in position. Something like this:

I dont want to use cv2.minMaxLoc() because there is multiple templates in image. I wrote a function that delete the close positions, but I want to know is there any straightforward solution for this problem? thanks.

Mescal

Can't you keep the instance with the best matching score ? – Cichocki

Because the template repeated multiple time in image, i can't use the best matching score and I should save locations with above a threshold. – Mescal

I mean the best among nearby instances (the distance limit can be as large as the template size). – Cichocki

I know, I'm doing something like that by myself. but its required to write a new function. I'm looking to something more simple. – Mescal

ok, seems that you don't need me. – Cichocki

@Mescal Please post your full images? – Disorient

See pyimagesearch.com/2021/03/29/… and pyimagesearch.com/2014/11/17/… – Khoury

Answer 1

One way to find multiple matches is to write over the found matches and run the match again. Edit: A better way to find multiple matches is to write over the results. In the first example we fill the matched part of results with zeroes (use ones for SQDIFF or CCORR_NORMED) , and then look for the next match in a loop.

import cv2
import numpy as np
import time

image = cv2.imread('smiley.png', cv2.IMREAD_COLOR )
template = cv2.imread('template.png', cv2.IMREAD_COLOR)

h, w = template.shape[:2]

method = cv2.TM_CCOEFF_NORMED

threshold = 0.90

start_time = time.time()

res = cv2.matchTemplate(image, template, method)

# fake out max_val for first run through loop
max_val = 1
while max_val > threshold:
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)

    if max_val > threshold:
        res[max_loc[1]-h//2:max_loc[1]+h//2+1, max_loc[0]-w//2:max_loc[0]+w//2+1] = 0   
        image = cv2.rectangle(image,(max_loc[0],max_loc[1]), (max_loc[0]+w+1, max_loc[1]+h+1), (0,255,0) )

cv2.imwrite('output.png', image)

input image:

use the eyes as template images (since there is more than one eye!)

output:

And here is the original way I did it by writing over the image. This way is way is much slower because we do n+1 matchTemplate operations for n matches. By one measurement, this technique is 1000 times slower.

import cv2
import numpy as np

image = cv2.imread('smiley.png', cv2.IMREAD_COLOR )
template = cv2.imread('template.png', cv2.IMREAD_COLOR)

h, w = template.shape[:2]

method = cv2.TM_CCOEFF_NORMED

threshold = 0.95

# fake out max_val for first run through loop
max_val = 1
while max_val > threshold:
    res = cv2.matchTemplate(image, template, method)
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)

    # using top ranked score, fill in that area with green
    image[max_loc[1]:max_loc[1]+h+1:, max_loc[0]:max_loc[0]+w+1, 0] = 0    # blue channel
    image[max_loc[1]:max_loc[1]+h+1:, max_loc[0]:max_loc[0]+w+1, 1] = 255  # green channel
    image[max_loc[1]:max_loc[1]+h+1:, max_loc[0]:max_loc[0]+w+1, 2] = 0    # red channel


cv2.imwrite('output.png', image)

output image:

Garrygarson

If matches occur near the edge of the image in the optimized function, max_loc[1]-h//2 and max_loc[0]-w//2 could be negative, which overwrites the wrong portion of the image and probably causes an infinite loop. Using max(..., 0) seems to fix the issue. – Huckleberry

Answer 2

While running bfris' method for my own dataset, I noticed that the while loop can stuck indefinitely in some special cases so here is my improved version:

import cv2
import numpy as np
import time

image = cv2.imread('smiley.png', cv2.IMREAD_COLOR )
template = cv2.imread('template.png', cv2.IMREAD_COLOR)

h, w = template.shape[:2]

method = cv2.TM_CCOEFF_NORMED

threshold = 0.90

start_time = time.time()

res = cv2.matchTemplate(image, template, method)

# fake out max_val for first run through loop
max_val = 1
prev_min_val, prev_max_val, prev_min_loc, prev_max_loc = None, None, None, None
while max_val > threshold:
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
    
    # Prevent infinite loop. If those 4 values are the same as previous ones, break the loop.
    if prev_min_val == min_val and prev_max_val == max_val and prev_min_loc == min_loc and prev_max_loc == max_loc:
        break
    else:
        prev_min_val, prev_max_val, prev_min_loc, prev_max_loc = min_val, max_val, min_loc, max_loc
    
    if max_val > threshold:
        # Prevent start_row, end_row, start_col, end_col be out of range of image
        start_row = max(0, max_loc[1] - h // 2)
        start_col = max(0, max_loc[0] - w // 2)
        end_row = min(res.shape[0], max_loc[1] + h // 2 + 1)
        end_col = min(res.shape[1], max_loc[0] + w // 2 + 1)

        res[start_row: end_row, start_col: end_col] = 0
        image = cv2.rectangle(image,(max_loc[0],max_loc[1]), (max_loc[0]+w+1, max_loc[1]+h+1), (0,255,0) )

cv2.imwrite('output.png', image)

Jacksmelt

Nice workaround. An equivalent way might be to use the clip function from NumPy: start_row = np.clip(max_loc[1]-h//2, 0, h). Your preference probably depends on which format you can make sense of when you come back to the code 1 year later. – Garrygarson

Answer 3

The same for TS, i've rewrite short python method and his improvement with my canges now we could use it for different methods. Maybe it could be helpfull for someone.

import * as cv from "opencv4nodejs-prebuilt-install";
import { Point2, Vec3 } from "opencv4nodejs-prebuilt-install";

enum MethodEnum {
  TM_CCOEFF = "TM_CCOEFF",
  TM_CCOEFF_NORMED = "TM_CCOEFF_NORMED",
  TM_CCORR = "TM_CCORR",
  TM_CCORR_NORMED = "TM_CCORR_NORMED",
  TM_SQDIFF = "TM_SQDIFF",
  TM_SQDIFF_NORMED = "TM_SQDIFF_NORMED",
}
type MethodName = `${MethodEnum}`;

export async function matchImagesByWriteOverFounded(
  haystack: cv.Mat,
  needle: cv.Mat,
  confidence: number = 0.99,
  matchedMethod: MethodName = MethodEnum.TM_SQDIFF_NORMED,
  debug: boolean = false
): Promise<Array<any>> {
  const h = needle.rows;
  const w = needle.cols;
  let minVal = 0;
  let maxVal = 1;
  let minMax, match;
  let matchedResults: Array<any> = [];
  let prevMinVal,
    prevMaxVal = 0;
  let prevMinLoc,
    prevMaxLoc = {};

  const isMethodTypeMaxOrMin =
    matchedMethod === MethodEnum.TM_SQDIFF_NORMED ||
    matchedMethod === MethodEnum.TM_SQDIFF;
  let locType = isMethodTypeMaxOrMin ? "minLoc" : "maxLoc";
  let confidentOffset =
    isMethodTypeMaxOrMin && confidence === 0.99 ? 0.008 : -0.19;
  const finalConfident =
    confidence + confidentOffset < 1
      ? confidence + confidentOffset
      : confidence;

  while (
    isMethodTypeMaxOrMin ? minVal <= finalConfident : maxVal > finalConfident
  ) {
    match = await haystack.matchTemplateAsync(needle, cv[`${matchedMethod}`]);
    minMax = await match.minMaxLocAsync();
    minVal = minMax.minVal;
    maxVal = minMax.maxVal;
    let { maxLoc, minLoc } = minMax;

    if (
      prevMinVal === minVal &&
      prevMaxVal === maxVal &&
      JSON.stringify(prevMinLoc) === JSON.stringify(minLoc) &&
      JSON.stringify(prevMaxLoc) === JSON.stringify(maxLoc)
    ) {
      break;
    } else {
      prevMinVal = minVal;
      prevMaxVal = maxVal;
      prevMinLoc = minLoc;
      prevMaxLoc = maxLoc;
    }

    if (
      isMethodTypeMaxOrMin ? minVal <= finalConfident : maxVal > finalConfident
    ) {
      const xL =
        (minMax[locType as keyof typeof minMax] as Point2).x >= 0
          ? (minMax[locType as keyof typeof minMax] as Point2).x
          : 0;
      const yL =
        (minMax[locType as keyof typeof minMax] as Point2).y >= 0
          ? (minMax[locType as keyof typeof minMax] as Point2).y
          : 0;
      const xR =
        (minMax[locType as keyof typeof minMax] as Point2).x + w + 1 >= 0
          ? (minMax[locType as keyof typeof minMax] as Point2).x + w + 1
          : 0;
      const yR =
        (minMax[locType as keyof typeof minMax] as Point2).y + h + 1 >= 0
          ? (minMax[locType as keyof typeof minMax] as Point2).y + h + 1
          : 0;

      haystack.drawFillPoly(
        [
          [
            new Point2(xL, yL),
            new Point2(xR, yL),
            new Point2(xR, yR),
            new Point2(xL, yR),
          ],
        ],
        new Vec3(0, 255, 0)
      );
      matchedResults.push(
        isMethodTypeMaxOrMin ? 1.0 - minMax.minVal : minMax.maxVal,
        (minMax[locType as keyof typeof minMax] as Point2).x,
        (minMax[locType as keyof typeof minMax] as Point2).y,
        needle.cols,
        needle.rows
      );
      if (debug) {
        cv.imshow("debug", haystack);
        cv.waitKey(0);
      }
    }
  }
  if (debug) {
    cv.imshow("debug", haystack);
    cv.waitKey(0);
  }
  return matchedResults;
}

Preliminaries