OpenCV template matching, multiple templates

Question

Im trying to make a bot for a game. Basically it picks up items from the ground, thing is these items look different sometimes for ex. the angle is different or they are lying on differentyl coloured ground etc. To make everything work I need multiple templates. Is there any way to do that? If you dont understand just tell me in the comments. Here is what I tried so far:

files = ["bones_{}.png".format(x) for x in range(6)]

    for i in range(6):
        img_gray = cv2.cvtColor(imageGrab(), cv2.COLOR_BGR2GRAY)
        f = str(files[i])
        template = cv2.imread(f, 0)
        w, h = template.shape[:: -1]
        res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
        threshhold = 0.70
        loc = np.where( res >= threshhold)

This works but It could be better. Do you have any ideas?

Listless

Answer 1

In your current code there are a lot of steps performed multiple times where once (per frame) suffices. You can gain efficiency by separating those out.

You're currently reloading the templates on every frame, this is very inefficient as you'll easily get 100+ loads per second. Instead create a list that holds the templates so they remain in memory. Accessing from memory is much faster than loading from disk.
You can do the same for the width/length of the templates, but it is not actually used in your code, so maybe you can skip it all together.
The threshold needs to be set only once.

templates = []
templ_shapes = []
threshold = 0.70

for i in range(6):
    templates.append(cv2.imread("bones_{}.png".format(i),0))
    templ_shapes.append(templates[i].shape[:: -1])

All the templates can be compared to the same screengrab, so you should take that outside the for loop. This is an easy but pretty big win. So on every frame, grab the screen once, and match all templates. For clarity and convenience you can put it in a function:

def doTemplateMatch():
    img_gray = cv2.cvtColor(imageGrab(), cv2.COLOR_BGR2GRAY)
    for template in templates: 
        res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
        loc = np.where( res >= threshold)

Dippy

Wow, this helped me so much! Thanks for all the help. Also can you upvote my question? Im on the edge of getting banned :/ – Listless

Answer 2

There’s a much better algorithm for matching with multiple templates. It’s faster than OpenCV. Check it out here, it’s called OpenFDCM. I just released it, it’s been months of development.

You can try it on Google Colab here.

pip install openfdcm

import openfdcm

templates = # A list of 4xN array where each array is a template represented as N lines [x1, y1, x2, y2]^T
scene = # A 4xM array representing the M scene lines

# Perform template matching
max_tmpl_lines, max_scene_lines = 4, 4  # Combinatory search parameters.
depth = 30              # The [0, pi] discretization.
coeff = 5.0             # A weighting factor to enhance the angular cost vs distance cost in FDCM algorithm.
scene_padding = 1.5     # A ratio to pad the scene images used in the FDCM algorithm, use if best match may appear on image boundaries.
distance_type = openfdcm.distance.L2 # or openfdcm.distance.L2_SQUARED or openfdcm.distance.L1
#num_threads = 4

threadpool = openfdcm.ThreadPool() # could pass num_threads here, but default is optimal
featuremap_params = openfdcm.Dt3CpuParameters(depth, coeff, scene_padding, distance_type)
search_strategy = openfdcm.DefaultSearch(max_tmpl_lines, max_scene_lines)
optimizer_strategy = openfdcm.BatchOptimize(10, threadpool)
matcher = openfdcm.DefaultMatch()
penalizer = openfdcm.ExponentialPenalty(tau=1.5)

# Build FDCm feature map and search
start_time = time.time()
featuremap = openfdcm.build_cpu_featuremap(scene, featuremap_params, threadpool)
raw_matches = openfdcm.search(matcher, search_strategy, optimizer_strategy, featuremap, templates, scene)
penalized_matches = openfdcm.penalize(penalizer, raw_matches, openfdcm.get_template_lengths(templates))
sorted_matches = openfdcm.sort_matches(penalized_matches)
search_time = time.time() - start_time

print(f"Template matching search completed in {search_time:.4f} seconds.")

best_match = sorted_matches[0]                 # Best match (lower score) is first
best_match_id = best_match.tmpl_idx
best_matched_tmpl = templates[best_match_id]
result_rotation = best_match.transform[0:2, 0:2]
result_translation = best_match.transform[0:2, 2]

Providenciaprovident

Looking at that repository, a major contributor is named "Jean-Christophe". If that's you, it would be a good idea to indicate the affiliation in your answer. See e.g. meta.stackexchange.com/q/229085/326191 – Fun

@DanMašek Sure! – Providenciaprovident