opencv - cropping handwritten lines (line segmentation)

Question

I'm trying to build a handwriting recognition system using python and opencv. The recognition of the characters is not the problem but the segmentation. I have successfully :

• segmented a word into single characters
• segmented a single sentence into words in the required order.

But I couldn't segment different lines in the document. I tried sorting the contours (to avoid line segmentation and use only word segmentation) but it didnt work. I have used the following code to segment words contained in a handwritten document , but it returns the words out-of-order(it returns words in left-to-right sorted manner) :

import cv2
import numpy as np
#import image
image = cv2.imread('input.jpg')
#cv2.imshow('orig',image)
#cv2.waitKey(0)

#grayscale
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
cv2.imshow('gray',gray)
cv2.waitKey(0)

#binary
ret,thresh = cv2.threshold(gray,127,255,cv2.THRESH_BINARY_INV)
cv2.imshow('second',thresh)
cv2.waitKey(0)

#dilation
kernel = np.ones((5,5), np.uint8)
img_dilation = cv2.dilate(thresh, kernel, iterations=1)
cv2.imshow('dilated',img_dilation)
cv2.waitKey(0)

#find contours
im2,ctrs, hier = cv2.findContours(img_dilation.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

#sort contours
sorted_ctrs = sorted(ctrs, key=lambda ctr: cv2.boundingRect(ctr)[0])

for i, ctr in enumerate(sorted_ctrs):
    # Get bounding box
    x, y, w, h = cv2.boundingRect(ctr)

    # Getting ROI
    roi = image[y:y+h, x:x+w]

    # show ROI
    cv2.imshow('segment no:'+str(i),roi)
    cv2.rectangle(image,(x,y),( x + w, y + h ),(90,0,255),2)
    cv2.waitKey(0)

cv2.imshow('marked areas',image)
cv2.waitKey(0)

Please note that i am able to segment all the words here but they appear out order.Is there any way to sort these contours in order of top to bottom

OR

segment the image into separate lines so that each line can be segmented into words using above code?

Foredo

why don't you sort it by x and y then? – Pianissimo

Actually i tried that but it just results in complete reordering of contours again. it doesnt return the contours in the correct order. I did left-to-right sorting followed by top-down sorting ,but contours are out of order. So i think each line would need to be separated. – Foredo

Making a list sorting contours based on x,y is the easiest i'd say. Add an image is you can. – Pyrophotometer

@SidharthRamesh hi there. Do you have any contact (WA, Telegram ) ? – Boor

@Link you could email me at [email protected] – Foredo

@SidharthRamesh How did separate word into single characters? Are there any references? – Christlike

@VishnuJayanand I updated the answer addressing your query, – Foredo

@SidharthRamesh Does reducing iterations work for separating handwritten characters? Also, how do you deal with text of different font sizes? – Christlike

@VishnuJayanand I dont understand your doubt on reducing iterations. This code will work for different font sizes because the cropped region for a line is calculated based on the largest character in that line. You can check the code with images having varying sized text. – Foredo

Answer 1

I got the required segmentation by making a change to the above code on the line:

kernel = np.ones((5,5), np.uint8)

I changed it to :

kernel = np.ones((5,100), np.uint8)

Now i get the outputs as following This also works with handwritten text images with lines that are not perfectly horizontal:

EDIT : For getting individual characters out of a word, do the following :

1. Resize the contour containing the word using the code as follows.

   im = cv2.resize(image,None,fx=4, fy=4, interpolation = cv2.INTER_CUBIC)
   

2. Apply same contour detection process as in line segmentation, but with a kernel of size (5,5), i.e :

   kernel = np.ones((5,5), np.uint8)
   img_dilation = cv2.dilate(im_th, kernel, iterations=1)
   

Foredo

Can you describe how to crop each bounding line into separate image. Any idea or resource. Your answer is helpful – Codger

@Sushant you mean saving each of the cropped contours into separate images ? just save each of the detected contours as an image. thats it. – Foredo