Segment an image using python and PIL to calculate centroid and rotations of multiple rectangular objects

Question

I am using python and PIL to find the centroid and rotation of various rectangles (and squares) in a 640x480 image, similar to this one

So far my code works for a single rectangle in an image.

import Image, math

def find_centroid(im):
    width, height = im.size
    XX, YY, count = 0, 0, 0
    for x in xrange(0, width, 1):
        for y in xrange(0, height, 1):
            if im.getpixel((x, y)) == 0:
                XX += x
                YY += y
                count += 1
    return XX/count, YY/count

#Top Left Vertex
def find_vertex1(im):
    width, height = im.size
    for y in xrange(0, height, 1):
        for x in xrange (0, width, 1):
            if im.getpixel((x, y)) == 0:
                X1=x
                Y1=y
                return X1, Y1

#Bottom Left Vertex
def find_vertex2(im):
    width, height = im.size
    for x in xrange(0, width, 1):
        for y in xrange (height-1, 0, -1):
            if im.getpixel((x, y)) == 0:
                X2=x
                Y2=y
                return X2, Y2

#Top Right Vertex
def find_vertex3(im):
    width, height = im.size
    for x in xrange(width-1, 0, -1):
        for y in xrange (0, height, 1):
            if im.getpixel((x, y)) == 0:
                X3=x
                Y3=y
                return X3, Y3

#Bottom Right Vertex
def find_vertex4 (im):
    width, height = im.size
    for y in xrange(height-1, 0, -1):
        for x in xrange (width-1, 0, -1):
            if im.getpixel((x, y)) == 0:
                X4=x
                Y4=y
                return X4, Y4

def find_angle (V1, V2, direction):
    side1=math.sqrt(((V1[0]-V2[0])**2))
    side2=math.sqrt(((V1[1]-V2[1])**2))
    if direction == 0:
        return math.degrees(math.atan(side2/side1)), 'Clockwise'
    return 90-math.degrees(math.atan(side2/side1)), 'Counter Clockwise'

#Find direction of Rotation; 0 = CW, 1 = CCW
def find_direction (vertices, C):
    high=480
    for i in range (0,4):
        if vertices[i][1]<high:
            high = vertices[i][1]
            index = i
    if vertices[index][0]<C[0]:
        return 0
    return 1

def main():
    im = Image.open('hopperrotated2.png')
    im = im.convert('1') # convert image to black and white
    print 'Centroid ', find_centroid(im)
    print 'Top Left ', find_vertex1 (im)
    print 'Bottom Left ', find_vertex2 (im)
    print 'Top Right', find_vertex3 (im)
    print 'Bottom Right ', find_vertex4 (im)
    C = find_centroid (im)
    V1 = find_vertex1 (im)
    V2 = find_vertex3 (im)
    V3 = find_vertex2 (im)
    V4 = find_vertex4 (im)
    vertices = [V1,V2,V3,V4]
    direction = find_direction(vertices, C)
    print 'angle: ', find_angle(V1,V2,direction)

if __name__ == '__main__':
  main()

Where I am having problems is when there is more than one object in the image.

I know PIL has a find_edges method that gives an image of just the edges, but I have no idea how to use this new edge image to segment the image into the separate objects.

from PIL import Image, ImageFilter

im = Image.open('hopperrotated2.png')

im1 = im.filter(ImageFilter.FIND_EDGES)
im1 = im1.convert('1')
print im1
im1.save("EDGES.jpg")

if I can use the edges to segment the image into individual rectangles then i can just run my first bit of code on each rectangle to get centroid and rotation.

But what would be better is to be able to use the edges to calculate rotation and centroid of each rectangle without needing to split the image up.

Everyone's help is greatly appreciated!

Shirt

I think you should take a look at scipy.ndimage specifically label to detect your rectangles, find_objects and center_of_mass. – Intercede

Answer 1

You need to identify each object before finding the corners. You only need the border of the objects, so you could also reduce your initial input to that. Then it is only a matter of following each distinct border to find your corners, the centroid is directly found after you know each distinct border.

Using the code below, here is what you get (centroid is the red point, the white text is the rotation in degrees):

Note that your input is not binary, so I used a really simple threshold for that. Also, the following code is the simplest way to achieve this, there are faster methods in any decent library.

import sys
import math
from PIL import Image, ImageOps, ImageDraw

orig = ImageOps.grayscale(Image.open(sys.argv[1]))
orig_bin = orig.point(lambda x: 0 if x < 128 else 255)
im = orig_bin.load()

border = Image.new('1', orig.size, 'white')
width, height = orig.size
bim = border.load()
# Keep only border points
for x in xrange(width):
    for y in xrange(height):
        if im[x, y] == 255:
            continue
        if im[x+1, y] or im[x-1, y] or im[x, y+1] or im[x, y-1]:
            bim[x, y] = 0
        else:
            bim[x, y] = 255

# Find each border (the trivial dummy way).
def follow_border(im, x, y, used):
    work = [(x, y)]
    border = []
    while work:
        x, y = work.pop()
        used.add((x, y))
        border.append((x, y))
        for dx, dy in ((1, 0), (-1, 0), (0, 1), (0, -1),
                (1, 1), (-1, -1), (1, -1), (-1, 1)):
            px, py = x + dx, y + dy
            if im[px, py] == 255 or (px, py) in used:
                continue
            work.append((px, py))

    return border

used = set()
border = []
for x in xrange(width):
    for y in xrange(height):
        if bim[x, y] == 255 or (x, y) in used:
            continue
        b = follow_border(bim, x, y, used)
        border.append(b)

# Find the corners and centroid of each rectangle.
rectangle = []
for b in border:
    xmin, xmax, ymin, ymax = width, 0, height, 0
    mean_x, mean_y = 0, 0
    b = sorted(b)
    top_left, bottom_right = b[0], b[-1]
    for x, y in b:
        mean_x += x
        mean_y += y
    centroid = (mean_x / float(len(b)), mean_y / float(len(b)))
    b = sorted(b, key=lambda x: x[1])
    curr = 0
    while b[curr][1] == b[curr + 1][1]:
        curr += 1
    top_right = b[curr]
    curr = len(b) - 1
    while b[curr][1] == b[curr - 1][1]:
        curr -= 1
    bottom_left = b[curr]

    rectangle.append([
        [top_left, top_right, bottom_right, bottom_left], centroid])


result = orig.convert('RGB')
draw = ImageDraw.Draw(result)
for corner, centroid in rectangle:
    draw.line(corner + [corner[0]], fill='red', width=2)
    cx, cy = centroid
    draw.ellipse((cx - 2, cy - 2, cx + 2, cy + 2), fill='red')
    rotation = math.atan2(corner[0][1] - corner[1][1],
            corner[1][0] - corner[0][0])
    rdeg = math.degrees(rotation)
    draw.text((cx + 10, cy), text='%.2f' % rdeg)

result.save(sys.argv[2])

Finespun

this is perfect! wow! only one issue I have is that when the rotation is clockwise I have to do 90-(subtract)angle. My original code does that by first checking to see if it is a clockwise rotation or counterclockwise by checking if the highest vertex is to the right or to the left of the centroid. Do you have a better solution to this? – Shirt

Looking at the code again, I see that when there is a clockwise rotation, the top_left corner won't be the actual top left corner of the figure. One way to detect that is checking whether the next two vertices are higher than the first one, in the code above this translates to if corner[0][1] > corner[1][1] and corner[0][1] > corner[2][1]: corner = corner[1:] + [corner[0]] (which would be added right after the start of the loop for corner, centroid in rectangle:). This reordering will give correct angles now (negative when clockwise). Is that what you had in mind ? – Finespun

That is what I had in mind, but adding that line in did not work. – Shirt

@Shirt Depending on how you see the rectangles, there are two distinct rotation angles so you have to decide which one you want to measure. If you have a solution that always work for you, I would stick to it. – Finespun

i had a solution...but it just seems too primitive to add to your code. i liked the idea of adding that line you suggested to make clockwise rotations negative. but it did not work :( – Shirt

@Shirt You didn't understand my previous remark: there are two ways to see the rectangle. There are two possible rotations, 62.1 degrees is one of them, 27.9 is another one, it depends on which axis you look at it. I don't see why you unaccepted my answer, that solves using only PIL to read the images as the request, and accepted something that uses ready libraries. – Finespun

@Shirt maybe what you want is simply to take into consideration the largest edge for getting the rotation degree: pastebin.com/VuRsC0qn – Finespun

you are absolutely right. thank you. I'm sorry I thought I could select both as the accepted answer. I will make yours the accepted because you are using PIL. sorry about that and thank you for all your help :) – Shirt

Oh, ok, sorry for the rudeness. Does the last suggestion help you in the angle measurement issue ? – Finespun

Answer 2

Here is an example of how you can do this by labelling the image, and then taking the centroid for the centers, this is all built in to ndimage in scipy (along with a bunch of other cool image things). For the angles, I've used the rectangle corner intercepts with the edges of the bounding slices.

import numpy as np
import scipy
from scipy import ndimage

im = scipy.misc.imread('6JYjd.png',flatten=1)
im = np.where(im > 128, 0, 1)
label_im, num = ndimage.label(im)
slices = ndimage.find_objects(label_im)
centroids = ndimage.measurements.center_of_mass(im, label_im, xrange(1,num+1))

angles = []
for s in slices:
    height, width = label_im[s].shape
    opp = height - np.where(im[s][:,-1]==1)[0][-1] - 1
    adj = width - np.where(im[s][-1,:]==1)[0][0] - 1
    angles.append(np.degrees(np.arctan2(opp,adj)))
print 'centers:', centroids
print 'angles:', angles

Output:

centers: [(157.17299748926865, 214.20652790151453), (219.91948280928594, 442.7146635321775), (363.06183745583041, 288.57169725293517)]
angles: [7.864024795499545, 26.306963825741803, 7.937188000622946]

Entophyte

You can also do it in Matlab/Mathematica in 2 or 3 lines. But that doesn't help a person in any way to learn something, I used only the very basic tools for that reason. – Finespun

@Finespun - I'd like to see that. It could probably also be written in many thousands of lines. But I have a preference for short solutions using the appropriate tools. For image analysis I usually turn to numpy, scipy, and opencv, they offer fast (to write, and run) and powerful solutions. And I think they are worth taking the time to investigate, for people interested in image analysis. And you are wrong... I learned something while coding this ;) – Entophyte

f = MorphologicalComponents[ColorNegate[Import["https://i.sstatic.net/6JYjd.png"]], 0.5]; cent = ComponentMeasurements [f, "Centroid"]; angle = ArcTan @@ (#[[1]] - #[[2]])/Degree & /@ ComponentMeasurements[f, "MinimalBoundingBox"][[All, 2, ;; 2]]; – Finespun

@Finespun - Thanks for that! turns out there is a similar function in opencv, that can e used for the angles, but getting the data into the right shape is a bit fiddly [cv2.minAreaRect(np.array(np.where(im[s]==1)).reshape(-1,1,2))[2] + 90 for s in slices] Now I've learnt two things :) – Entophyte

@Entophyte - thank you for this beautiful code! You and mmgp have been more than helpful. The only problem i am having now is that objects that are rotated clockwise do not show the right angle. I need to be able to distinguish the direction of rotation so that I can do 90 degrees subtracted by the output angle to get the correct angle for the clockwise rotation. – Shirt

@Entophyte - here is another image I am using. the angle on the clockwise rotated rectangle outputs as 62 degrees. but it should be 90-62. what is a good way of fixing this? – Shirt

@Shirt - Glad to help :) I see your issue. Currently the code measures the angle between the horizontal and the bottom right side of each rectangle. But you want it to measure the angle between the longest side of each rectangle and the horizontal. You should be able to easily adapt the above code to test whether the bottom left side or bottom right side of each rectangle is longest, and then use this side to calculate the angle. Hope that helps, and good luck ;) – Entophyte

@Entophyte - thank you very much. your code is very interesting and I am trying to learn what you did. I am still very new to image processing (as you can tell) so help from people like you who have obviously mastered these techniques allows me to take huge leaps ahead. – Shirt