You can compare the area of the mask to its perimeter using the following formula
ratio = 4 * pi * Area / ( Perimeter^2 )
For circles this ration should be very close to one, for other shapes it should be significantly lower.
See this tutorial for an example.
The rationale behind this formula: circles are optimal in their perimeter-area ratio - max area for given perimeter. Given Perimeter, you can estimate radius of equivalent circle by Perimeter = 2*pi*R
, using this estimated R
you can compute the "equivalent circle area" using eqArea = pi*R^2
. Now you only need to check the ratio between the actual area of the shape and the "equivalent area" computed.
Note: since Area
and Perimeter
of objects in mask are estimated based on the pixel-level discretization these estimates may be quite crude especially for small shapes. Consider working with higher resolution masks if you notice quantization/discretization errors.