how to segment the connected area based on depth color in opencv

Question

I have a picture like , which i need to segment the picture into 8 blocks.

I have tried this threshold method

img_gray = cv2.imread(input_file,cv2.IMREAD_GRAYSCALE)
ret,thresh = cv2.threshold(img_gray,254,255,cv2.THRESH_BINARY) =
kernel = np.array(cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3), (-1, -1)))
img_open = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel)
cv2.imshow('abc',img_open)
ret1,thresh1 = cv2.threshold(img_open,254,255,cv2.THRESH_BINARY_INV) #
contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_CCOMP ,cv2.CHAIN_APPROX_NONE)

for i in range(len(contours)):
    if len(contours[i]) > 20:
        x, y, w, h = cv2.boundingRect(contours[i])
        cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2)
        print (x, y),(x+w, y+h)

after the thresholding

the end result is some blocks connected together are formed into a large segment, which is not what I hoped. Any other ways to get it around

Furey

I'm not familiar with openCV, but I feel the problem lies in the fact that you are doing a binary threshold, that is either it's there or not. But the results you need are trinary - both rows and then the background. Rather than using cv2.THRESH_BINARY try cv2.THRESH_TRUNC to first omit the background, and then a binary comparison to separate the darker front row from the lighter rear row. Finally retrieve the contours for both, combine them, and you should in theory end up with 8 regions. – Optometer

Have you tried meanshift segmentation based on location+depth? – Edirne

@Edirne I will look into the meanshift segmentation – Furey

You might want to look into Super Pixel – Rosariorosarium

Since you mentioned k-means, do you know the number of cars in the picture beforehand? – Ambrotype

@Ambrotype Yes, I can know the number of cars beforehand – Furey

@Ambrotype I don't think k-means would work here: k-means assumes the clusters are "round" (i.e., extends evenly at all axes), which is not the case here. One would need to consider GMM (fitting a "covariance" matrix for each cluster to compensate for the un-even distribution). – Edirne

@Furey if you know the number of clusters/cars in advance, have you considered using Normalized Cuts? – Edirne

@Edirne I agree using k-means might be quite tricky. However, the reason I asked is that Shai mentioned about depth image having nice gradients, and if you know the number of cars it can really help to just use simple thresholds... Maybe a bit too naive but I can see how knowing the number of cars beforehand can help improve rick-m's answer. – Ambrotype

Answer 1

I'll try and give you a sketch of an algorithm that separates the cars based on depth gradients. Alas, simply looking at the contour of large depth gradients, the cars are not perfectly separated, therefore, some "refinement" of the boundary contour is required. Once the contours are complete, a simple connected component clustering is sufficient to separate the cars.

Here's my code (in Matlab, but I'm quite certain it's not too complex to find opencv equivalent functions):

img = imread('https://i.sstatic.net/8lJw8.png');  % read the image
depth = double(img(:,:,1));
depth(depth==255)=-100;  % make the background VERY distinct
[dy dx] = gradient(depth);  % compute depth gradients
bmsk = sqrt(dx.^2+dy.^2) > 5;  % consider only significant gradient
% using morphological operations to "complete" the contours around the cars
bmsk = bwmorph( bwmorph(bmsk, 'dilate', ones(7)), 'skel'); 

% once the contours are complete, use connected components
cars = bwlabel(~bmsk,4);  % segmentation mask
st = regionprops(cars, 'Area', 'BoundingBox');
% display the results
figure;
imshow(img);
hold all;
for ii=2:numel(st),  % ignore the first segment - it's the background
    if st(ii).Area>200, % ignore small regions as "noise"
        rectangle('Position',st(ii).BoundingBox, 'LineWidth', 3, 'EdgeColor', 'g');
    end;
end;

The output is

And

Not perfect, but brings you close enough.

Further reading:

• bwmorph: to perform morphological operations.
• bwlabel: to output a segmentation mask (labeling) of the connected components.
• regionprops: compute statistics (e.g., area and bounding box) for image regions.

---

Coming to think of it, depth has such nice gradients, you can threshold the depth gradient and get nice connected components.

Edirne

depth = double(img(:,:,1)); depth(depth==255)=-100; In the program I don't understand this two, img(:,:,1) means red channel of the img, we use double to change its data type to double from uint8. When we write depth(depth == 255) = -100 means we replace all 255 values in depth replaced by -100. The question is what happened and why we do this? – Furey

Answer 2

Naive Approach (But it works)

Step 1: After reading the image in gray scale, threshold to get bottom cars.

ret1, car_thresh1 = cv2.threshold(cars, 191, 254, 0)

which gave me this.

Step 2: Subtract this image from the main image

car_thresh2 = car_thresh1 - cars

which gave me this.

Step 3: Threshold the subtracted image

ret3, cars_thresh3 = cv2.threshold(car_thresh2, 58, 255, 0)

which gave me this

Then I simply did what you did for extracting and drawing contours in the carsTop and carsBottom and this is the result.

Rosariorosarium

this way would work for frontal picture, but for the last image, it would fail – Furey

Can you add the last image without the rectangles drawn? – Rosariorosarium

So I tried with that image as well, although it does work by simple thresholding and subtraction, the intuition of the threshold values isn't automatic. I will try and come up with a better, more robust answer. This was, as mentioned in the answer, a Naive approach. – Rosariorosarium

I thought It could be working with some segmentation method like kmeans or graphcut, I will try as well – Furey