How to overlay segmented image on top of main image in python

Question

I have an image in RGB and another segmented image in which the pixels have 3 values(segmented image). I want to overlay the segmented image on top of the main image as the segmented areas make contours over the main image such as image below. Here the value of the segmented image pixels are 0, 1 and 2. The red contour shows the contour of pixels with value1 , the yellow contour shows the contour of pixels with 2 value and the background pixel value is 0.

the image is from the paper "Dilated-Inception Net: Multi-Scale FeatureAggregation for Cardiac Right VentricleSegmentation"

Here is an example of a segmented image.

segmented image

The background image can be any image. I only need these rectangle counters appear on the background image as two contours similar to red and yellow lines above. So, the output will be similar to the image below.

output image

sorry as I draw rectangles by hand they are not exact. I just would like to give you an insight about the output.

Briquet

Your question, if there is one, is quite unclear. You said you have an RGB image and another segmented image. Can we have both please? Then what do you want done with the rectangles in your final image exactly? Thank you. – Buiron

The background image can be any RGB image that is why I did not add any as main image. I only need the counters of the rectangles in the second image overlay on the background(main) image in a way that the red and yellow lines in the first image overlay the background image of heart CT image. – Briquet

How can I know what your segmented image looks like? Have you got an SVG path of a red line? Have you got a transparent background with a red line outlining a shape? Have you got a solid image where each tissue/object has a solid colour? If you would like some assistance, it's generally a good idea to make it easy for folks to help you. Also, what are "rectangle counters" please? – Buiron

I have added the output I need . Thanks for help. – Briquet

rectangle contours are the red and yellow lines in the output image. they should be drawn over the pixels occupied by the rectangles in the segmented image. – Briquet

Answer 1

I had a go at this using four different methods:

• OpenCV
• PIL/Pillow and Numpy
• command-line with ImageMagick
• morphology from skimage

---

Method 1 - OpenCV

• Open segmented image as greyscale
• Open main image as greyscale and make colour to allow annotation
• Find the contours using cv2.findContours()
• Iterate over contours and use cv2.drawContours() to draw each one onto main image in colour according to label in segmented image.

Documentation is here.

So, starting with this image:

and this segmented image:

which looks like this when contrast-stretched and the sandwich is labelled as grey(1) and the snout as grey(2):

Here's the code:

#!/usr/bin/env python3

import numpy as np
import cv2

# Load images as greyscale but make main RGB so we can annotate in colour
seg  = cv2.imread('segmented.png',cv2.IMREAD_GRAYSCALE)
main = cv2.imread('main.png',cv2.IMREAD_GRAYSCALE)
main = cv2.cvtColor(main,cv2.COLOR_GRAY2BGR)

# Dictionary giving RGB colour for label (segment label) - label 1 in red, label 2 in yellow
RGBforLabel = { 1:(0,0,255), 2:(0,255,255) }

# Find external contours
_,contours,_ = cv2.findContours(seg,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)

# Iterate over all contours
for i,c in enumerate(contours):
    # Find mean colour inside this contour by doing a masked mean
    mask = np.zeros(seg.shape, np.uint8)
    cv2.drawContours(mask,[c],-1,255, -1)
    # DEBUG: cv2.imwrite(f"mask-{i}.png",mask)
    mean,_,_,_ = cv2.mean(seg, mask=mask)
    # DEBUG: print(f"i: {i}, mean: {mean}")

    # Get appropriate colour for this label
    label = 2 if mean > 1.0 else 1
    colour = RGBforLabel.get(label)
    # DEBUG: print(f"Colour: {colour}")

    # Outline contour in that colour on main image, line thickness=1
    cv2.drawContours(main,[c],-1,colour,1)

# Save result
cv2.imwrite('result.png',main) 

Result:

---

Method 2 - PIL/Pillow and Numpy

• Open segmented image and find unique colours
• Open main image and desaturate
• Iterate over each unique colour in list
• ... Make all pixels that colour white and all others black
• ... Find edges and use edges as mask to draw colour on main image

Here's the code:

#!/usr/bin/env python3

from PIL import Image, ImageFilter
import numpy as np

def drawContour(m,s,c,RGB):
    """Draw edges of contour 'c' from segmented image 's' onto 'm' in colour 'RGB'"""
    # Fill contour "c" with white, make all else black
    thisContour = s.point(lambda p:p==c and 255)
    # DEBUG: thisContour.save(f"interim{c}.png")

    # Find edges of this contour and make into Numpy array
    thisEdges   = thisContour.filter(ImageFilter.FIND_EDGES)
    thisEdgesN  = np.array(thisEdges)

    # Paint locations of found edges in color "RGB" onto "main"
    m[np.nonzero(thisEdgesN)] = RGB
    return m

# Load segmented image as greyscale
seg = Image.open('segmented.png').convert('L')

# Load main image - desaturate and revert to RGB so we can draw on it in colour
main = Image.open('main.png').convert('L').convert('RGB')
mainN = np.array(main)

mainN = drawContour(mainN,seg,1,(255,0,0))   # draw contour 1 in red
mainN = drawContour(mainN,seg,2,(255,255,0)) # draw contour 2 in yellow

# Save result
Image.fromarray(mainN).save('result.png')

You'll get this result:

---

Method 3 - ImageMagick

You can also do the same thing from the command-line without writing any Python, and just using ImageMagick which is installed on most Linux distros and is available for macOS and Windows:

#!/bin/bash

# Make red overlay for "1" labels
convert segmented.png -colorspace gray -fill black +opaque "gray(1)" -fill white -opaque "gray(1)" -edge 1 -transparent black -fill red     -colorize 100% m1.gif
# Make yellow overlay for "2" labels
convert segmented.png -colorspace gray -fill black +opaque "gray(2)" -fill white -opaque "gray(2)" -edge 1 -transparent black -fill yellow  -colorize 100% m2.gif
# Overlay both "m1.gif" and "m2.gif" onto main image
convert main.png -colorspace gray -colorspace rgb m1.gif -composite m2.gif -composite result.png

---

Method 4 - Morphology from skimage

Here I am using morphology to find black pixels near 1 pixels and black pixels near 2 pixels.

#!/usr/bin/env python3

import skimage.filters.rank
import skimage.morphology
import numpy as np
import cv2

# Load images as greyscale but make main RGB so we can annotate in colour
seg  = cv2.imread('segmented.png',cv2.IMREAD_GRAYSCALE)
main = cv2.imread('main.png',cv2.IMREAD_GRAYSCALE)
main = cv2.cvtColor(main,cv2.COLOR_GRAY2BGR)

# Create structuring element that defines the neighbourhood for morphology
selem = skimage.morphology.disk(1)

# Mask for edges of segment 1 and segment 2
# We are basically looking for pixels with value 1 in the segmented image within a radius of 1 pixel of a black pixel...
# ... then the same again but for pixels with a vaue of 2 in the segmented image within a radius of 1 pixel of a black pixel
seg1 = (skimage.filters.rank.minimum(seg,selem) == 0) & (skimage.filters.rank.maximum(seg, selem) == 1)
seg2 = (skimage.filters.rank.minimum(seg,selem) == 0) & (skimage.filters.rank.maximum(seg, selem) == 2)

main[seg1,:] = np.asarray([0, 0,   255]) # Make segment 1 pixels red in main image
main[seg2,:] = np.asarray([0, 255, 255]) # Make segment 2 pixels yellow in main image

# Save result
cv2.imwrite('result.png',main) 

---

Note: JPEG is lossy - do not save your segmented image as JPEG, use PNG or GIF!

Keywords: Python, PIL, Pillow, OpenCV, segmentation, segmented, labelled, image, image processing, edges, contours, skimage, ImageMagick, scikit-image, morphology, rank, ranking filter, pixel adjacency.

Buiron

Answer 2

These are quick one-liners that automatically choose colors for the category/class integer values and execute the overlay onto the original image.

Color entire segmentation area:

from skimage import color
result_image = color.label2rgb(segmentation_results, input_image)

Color contours of segmentation areas:

from skimage import segmentation
result_image = segmentation.mark_boundaries(input_image, segmentation_results, mode='thick')

Kyte

Answer 3

If semi-transparent segmentation masks are to be displayed on top of the image, skimage has a built-in label2rgb() function that colorizes by a label channel:

Input Image

from skimage import io, color
import matplotlib.pyplot as plt
import numpy as np

seg = np.zeros((256,256)) # create a matrix of zeroes of same size as image
seg[gt > 0.95] = 1   # Change zeroes to label "1" as per your condition(s)
seg[zz == 255] = 2   

io.imshow(color.label2rgb(seg,img,colors=[(255,0,0),(0,0,255)],alpha=0.01, bg_label=0, bg_color=None))
plt.show()

Petrel

That's very nice, but it converts the input image to gray-scale. Do you know of any alternatives if the image in question is RGB? – Hadley

The argument img accepts both RGB and grayscale images. So it should work fine. Documentation confirms it - scikit-image.org/docs/dev/api/… – Petrel

It does accept RGB images, but then it converts to gray-scale before applying the segmentation overlay. I wish to see the original, colored image behind the overlay. Edit: I thought this saturation arg might be the key, but it's only present on skimage's dev branch, which doesn't seem to install on colab. – Hadley

This is helpful! But the scale makes no sense with respect to the colors shown, I don't know how to tweak that (a bit new to image analysis + python), if anyone has pointers that would be super helpful – Sequela