Create Bounding Boxes from Image Labels

Question

I have a 2D color image and a label image (projection of labels)

The output of label image is as follows:

[[16 16 16 ... 16 16 16 ]
 [16 16 16 ... 16 16 16 ]
 [16 16 16 ... 16 16 16 ]
 ...
 [ 2  2  2 ...  2  2  2 ]
 [ 2  2  2 ...  2  2  2 ]
 [ 2  2  2 ...  2  2  2 ]]

How do I draw bounding boxes around all the objects (represented by labels) in the original 2D color image?

Larger

Are you doing pixel-level classification? – Carleecarleen

Yes, these are instance segmentation labels. Each pixel corresponds to a label (for example: 1 is chair, 2 is bed and so on) – Larger

How are you displaying the images? – Carleecarleen

Using cv2.imread and cv2.imshow... I would like to display the ground truth bounding boxes (I only have ground truth instance segmentation) on the color image using cv2.rectangle but I don't know how to compute bounding box corner coordinates. – Larger

Can you compute the points that would belong to each bounding box? If so, you can get the corner coordinates from there. Determining what defines the bounding boxes is something you’ll have to answer by looking at your data. – Carleecarleen

I'm struggling with that part, any help would be appreciated – Larger

Answer 1

This task can be easily done using NumPy's boolean array indexing and OpenCV's boundingRect function.

From here, I took this image

and this segmentation mask

The mask is an indexed image, which OpenCV has problems with, see also here. That's why, we'll also use Pillow for this task.

Here's the code:

import cv2
import numpy as np
from PIL import Image

# Read color image
img = cv2.imread('0.jpg')

# Read mask; OpenCV can't handle indexed images, so we need Pillow here
# for that, see also: https://mcmap.net/q/1781956/-opencv-reads-an-image-as-a-3-channel-image-while-pil-reads-the-same-image-as-a-1-channel-image/11089932
mask = np.array(Image.open('0_mask.png'))

# Iterate all colors in mask
for color in np.unique(mask):

    # Color 0 is assumed to be background or artifacts
    if color == 0:
        continue

    # Determine bounding rectangle w.r.t. all pixels of the mask with
    # the current color
    x, y, w, h = cv2.boundingRect(np.uint8(mask == color))

    # Draw bounding rectangle to color image
    out = cv2.rectangle(img.copy(), (x, y), (x+w, y+h), (0, int(color), 0), 2)

    # Show image with bounding box
    cv2.imshow('img_' + str(color), out)

# Show mask
cv2.imshow('mask', mask)
cv2.waitKey(0)
cv2.destroyAllWindows()

These are the outputs:

I decided to output several images, because for the given image, the bounding boxes heavily overlap. To draw all rectangles to the same image, just replace the corresponding rectangle command by

img = cv2.rectangle(img, (x, y), (x+w, y+h), (0, int(color), 0), 2)

----------------------------------------
System information
----------------------------------------
Platform:    Windows-10-10.0.16299-SP0
Python:      3.8.5
NumPy:       1.19.2
OpenCV:      4.4.0
Pillow:      7.2.0
----------------------------------------

Fibrinous

Thank you, that's very helpful! – Larger