How to test accuracy of segmentation algorithm?

Question

I am dealing with a image classification problem. Before classification, images should be segmented. I tried several methods. My question is "how can i test accuracy of segmentation ?". I plan to compare final binary image with correct binary image based on pixel differences in order to get a success rate. Is there a more efficient way to compare edges of two binary image, instead of this?

Exocentric

Answer 1

Measuring the quality of image segmentation is a topic well studied in the computer vision community.

You can see this method that is suitable for binary segmentations. There is also this method for multiple segments and also for boundary accuracy.

Garrygarson

Answer 2

A usual approach is to use the ratio of the total area of the correct position of the object compared to the area of the detected object that falls into the correct position.

If your areas are not uniform, it will be something like (pixels in the detected area that match the ground truth)/total number of pixels in the ground truth segmentation.

in the image below: count(gray)/(count(black+gray))

A measure you should consider is also a ratio of the detection area compared to the ground truth area, because you may have a detection that covers the whole image, and have a score of 100% accuracy on the above formula.

Caucasia

would just like to add this link (to one of the standard segmentation datasets):eecs.berkeley.edu/Research/Projects/CS/vision/bsds basically it might give the OP a hands-on idea of the segmentation boundary and benchmark results comparison that is associated with segmentation evaluations – Nastassia

This measure is called Dice score. – Doud

This is not dice. Dice would be 2 * count(intersection)/(count(ground_truth) + count(detection)) – Systematist

Answer 3

I think multiple measures should be used when you want to evaluate your segmentation result. The accuracy ( the ratio of the correctly segmented area over the ground truth) is not enough. Because your segmentation may also cover the area that is not in the ground truth. So, I suggest you can use the following measures to evaluate your segmentation result:

1. True positive rate: the correctly segmentation area over all the area you segmented.
2. False positive rate: the area that is not in the ground truth but that is in your result over all the area you segmented.
3. Accuracy
4. F1 score: an integrated measure (please see: http://en.wikipedia.org/wiki/F1_score)

Ingles

Answer 4

You can use jaccard_similarity_score as shown here: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_similarity_score.html But for images needs flattening the images for converting it into 1-D

Sachikosachs

Answer 5

And how happy would you be if the ground truth object is detected in 1000 little segments that perfectly cover the area?

Slosh