finding convexity defects in opencv? [crashes depending on the given input image..]

Question

I have a program that calculates the convex hull of an image. I'm trying to use this information in order to count the number of fingers that are present in an input image. From some surfing I found out that the way to do this (count fingers) is by

1. Finding contours
2. Convex Hull
3. Convexity defects

But I'm having trouble using the convexity defects function. It compiles fine but at runtime the program crashes with certain input images but not with others and I can't seem to figure out why.

These are the input images

1. this image causes a crash
2. but this does not.
3. this also causes a crash even though its similar to the above

code..

#include <opencv/cv.h>
#include <opencv/highgui.h>
#include <opencv/cxcore.h>
#include <stdio.h>

#define CVX_RED     CV_RGB(0xff,0x00,0x00)
#define CVX_GREEN   CV_RGB(0x00,0xff,0x00)
#define CVX_BLUE    CV_RGB(0x00,0x00,0xff)

int main(int argc, char* argv[]) {

  cvNamedWindow( "original", 1 );
  cvNamedWindow( "contours", 1 );
  cvNamedWindow( "hull", 1 );
  IplImage* original_img = NULL;

  original_img = cvLoadImage("img.jpg", CV_LOAD_IMAGE_GRAYSCALE );

  IplImage* img_edge = cvCreateImage( cvGetSize(original_img), 8, 1 );
  IplImage* contour_img = cvCreateImage( cvGetSize(original_img), 8, 3 );
  IplImage* hull_img = cvCreateImage( cvGetSize(original_img), 8, 3 );

  cvThreshold( original_img, img_edge, 128, 255, CV_THRESH_BINARY );

  CvMemStorage* storage = cvCreateMemStorage();
  CvSeq* first_contour = NULL;

  int Nc = cvFindContours(
     img_edge,
     storage,
     &first_contour,
     sizeof(CvContour),
     CV_RETR_LIST // Try all four values and see what happens
  );

  for( CvSeq* c=first_contour; c!=NULL; c=c->h_next ) {
     cvCvtColor( original_img, contour_img, CV_GRAY2BGR );
     cvDrawContours(
        contour_img,
        c,
        CVX_RED,  
        CVX_BLUE,
        0,     
        2,
        8
     );
  }

  //----------------------------------------------------------------------Convex Hull

  CvMemStorage* hull_storage = cvCreateMemStorage();
  CvSeq* retHulls = NULL;

  for(CvSeq* i = first_contour; i != NULL; i = i->h_next){
    retHulls = cvConvexHull2(i,hull_storage,CV_CLOCKWISE,0); 
    // with 1 it draws the Hull image but not with 0..?
    // however it needs to be 0 for convexitydefects to work?
  }

  printf(" %d elements:\n", retHulls->total );

  // drawing hull

  for( CvSeq* j=retHulls; j!=NULL; j=j->h_next ) {
    cvCvtColor( original_img, hull_img, CV_GRAY2BGR );
    cvDrawContours(
        hull_img,
        j,
        CVX_RED,  
        CVX_BLUE,
        0,        
        2,
        8
     );  

  }


  //----------------------------------------------------------------------Convexity Defects??

  CvMemStorage* convexStorage = cvCreateMemStorage();
  CvSeq* defect = NULL;
  defect = cvConvexityDefects(first_contour,retHulls, convexStorage);
  printf(" %d defect:\n", defect->total );


  cvShowImage( "contours", contour_img );
  cvShowImage( "original", original_img );
  cvShowImage( "hull", hull_img );
  cvWaitKey(0);
  cvDestroyWindow( "contours" );
  cvDestroyWindow( "original" );
  cvDestroyWindow( "hull" );
  cvReleaseImage( &original_img );
  cvReleaseImage( &contour_img );
  cvReleaseImage( &hull_img );
  cvReleaseImage( &img_edge );
  return 0;
}

Sassaby

Have you tried using the C++ interface to check if the problem persists or it has to be C? Just asking, because you tagged C++ also. – Hijack

@Bob Hello, I don't think there is a c++ interface for the convexity function. I added the tag for view. – Sassaby

cv::convexHull and cv::convexityDefects exist in 2.4.2. I don't know about 2.3 though. – Hijack

Ah! cheers Bob, I'll have a look and see. Thanks. – Sassaby

My advice for you would be to upgrade to the latest opencv version and to use all C++ and not use those deprecated C stuff. IplImage has been deprecated, now Mat is used instead. It is not related to your problem but you might be bumping into some bug that has been fixed. – Kilimanjaro

@RuiMarques Keep in mind that the C++ API uses the C API to do some stuff behind your back. ;) Upgrading might not solve the problem, but it's worth to try. – Filter

an example of using C++ API buried in one weird app (with OpenCV 2.4.x) – Augment

Answer 1

cvConvexityDefects expects the convexHull sequence (second argument) to contain indices into the contour sequence (first argument):

Convex hull obtained using ConvexHull2 that should contain pointers or indices to the contour points, not the hull points themselves

1. In the most trivial case, where cvFindContours returns a single simple contour (your second image) you got lucky and your code would supply the correct sequence as the first parameter.

2. In case cvFindContours finds holes in the contour (your third image), or if there are several simple contours or contours with holes (your first image) your code:

   1. finds a convex hull of each contour in turn, but only remembers the last one (since each iteration of the loop overwrites retHulls variable)

   2. passes the whole hierarchy of the contours, which doesn't correspond to indices in retHulls, to cvConvexityDefects as the first argument.

Instead, you should have:

1. passed CV_RETR_EXTERNAL to the cvFindContour to only get the outer contours (you don't care for defects of holes)

2. moved the cvConvexityDefects inside the last loop.

Something like:

  /* ... */

  if (argc < 2) {
      std::cerr << "Usage: convexity IMAGE\n";
      exit(1);
  }

  cvNamedWindow( "original", 1 );
  cvNamedWindow( "contours", 1 );
  cvNamedWindow( "hull", 1 );
  IplImage* original_img = NULL;

  original_img = cvLoadImage(argv[1], CV_LOAD_IMAGE_GRAYSCALE );

  IplImage* img_edge = cvCreateImage( cvGetSize(original_img), 8, 1 );
  IplImage* contour_img = cvCreateImage( cvGetSize(original_img), 8, 3 );
  IplImage* hull_img = cvCreateImage( cvGetSize(original_img), 8, 3 );

  cvThreshold( original_img, img_edge, 128, 255, CV_THRESH_BINARY );

  CvMemStorage* storage = cvCreateMemStorage();
  CvSeq* first_contour = NULL;

  int Nc = cvFindContours(
     img_edge,
     storage,
     &first_contour,
     sizeof(CvContour),
     CV_RETR_EXTERNAL // Try all four values and see what happens
  );

  cvCvtColor( original_img, contour_img, CV_GRAY2BGR );
  for( CvSeq* c=first_contour; c!=NULL; c=c->h_next ) {
     cvDrawContours(
        contour_img,
        c,
        CVX_RED,
        CVX_BLUE,
        0,
        2,
        8
     );
  }
  cvShowImage( "contours", contour_img );

  //----------------------------------------------------------------------Convex Hull
  //-------------------------------------------------------------------Convex Defects

  CvMemStorage* hull_storage = cvCreateMemStorage();
  CvSeq* retHulls = NULL;

  cvCvtColor( original_img, hull_img, CV_GRAY2BGR );
  for(CvSeq* i = first_contour; i != NULL; i = i->h_next){
    retHulls = cvConvexHull2(i,hull_storage,CV_CLOCKWISE,0);
    printf(" %d elements:\n", retHulls->total );

    CvSeq* defect = NULL;
    defect = cvConvexityDefects(i,retHulls, NULL); // reuse storage of the contour
    printf(" %d defect:\n", defect->total );

    // drawing hull.... you can't use the one returned above since it only
    // contains indices
    retHulls = cvConvexHull2(i,hull_storage,CV_CLOCKWISE,1);
    cvDrawContours(
        hull_img,
        retHulls,
        CVX_RED,
        CVX_BLUE,
        0,
        2,
        8
     );
  }

  cvShowImage( "hull", hull_img );
  /* ... */

Augment

Regardless of this solving the problem or not, the fact is that the function shouldn't crash or freeze with invalid parameters. This is a real issue and it should be addressed on OpenCV's issue tracker. Your answer seems nice, though. When the OP test this I'll up vote it. – Filter

I agree. OpenCV is often poorly documented, fails silently or with cryptic assertion errors. Half of the times I figure out details by trial and error and the rest - by reading the sources carefully. – Augment

Answer 2

Running your application with the problematic images freezes it, but I see no crash with OpenCV 2.4.2, and the problem is really happening at cvConvexityDefects(), according to gdb:

(gdb) bt
#0  0x00000001002b1491 in cvConvexityDefects ()
#1  0x0000000100001a8d in main ()

Can't tell you why, though. Since the parameters seem to be OK, you might want to register a new issue here.

Filter

I'm using opencv 2.3 at the moment, are you saying that under 2.4.2 all images work? or still same issue with the problematic images? – Sassaby

On 2.4.2 your program hangs at cvConvexityDefects() instead of crashing. It's just a different behavior from what you've observed, but it still indicates that the problem is there. – Filter

Answer 3

Using Python 3 and opencv 4.5.1, I encountered a similar issue where the indices from convexHull() were 'not monotonic'. I found that for some reason the indices were returned from this function out of order.

To resolve the issue, I simply sorted the numpy array in descending order before passing the indices to convexityDefects().

hull = cv2.convexHull(contours, returnPoints=False)
hull[::-1].sort(axis=0)
defects = cv2.convexityDefects(contours, hull)

Semi

Answer 4

I've seen this problem before and that's when I use MAT with OpenCV 4.4.0 and I guess the crash happened with this error.

"The convex hull indices are not monotonous, which can be in the case when the input contour contains self-intersections in function 'convexityDefects'.

The solution was easy as you read the crash report in details, Which is that the indices are not sorted in order, and may be that's because there is some self-intersections in the contour, (or it might be a glitch since ver 2.1 fixed on ver 3.2 then back again on the 4.4.0 Check:
https://github.com/opencv/opencv/issues/4539

So in order not to wonder about this issue a lot, I have solve it by resorting the hull indices it self before extracting the defects from the contours.

As you see the hull indices is sorted upside down as if the hull cell size is 6 it means the indices in this cell will be:

[0] = 5
[1] = 4
[2] = 3
[3] = 2
[4] = 1
[5] = 0

But for the intersection or some other reasons it may not be sorted as it should like for example:

[0] = 6
[1] = 5
[2] = 4
[3] = 2
[4] = 1
[5] = 0

So all we need to do is to resort it check this code

vector <vector <Vec4i> > defects(contour.size());  // Defects Vectors
vector <vector <cv::Point> > hullsP(contour.size()); // Hulls contour points
vector <vector <int> > hullsI(contour.size()); // Indices to hulls contour points

for(int i = 0; i < contour.size(); i++)
{
    convexHull(contour[i], hullsP[i], CV_CLOCKWISE, false);
    convexHull(contour[i], hullsI[i], CV_CLOCKWISE, false);
    
    for(size_t k =0; k < hullsI[i].size(); k++) //Here we resort the indices
    {
        if (hullsI[i].size() > 0)
        {
            hullsI[i][k] = (int)((hullsI[i].size() - k)-1);
        }
    }
    
    if(hullsI[i].size() > 3 ) // You need more than 3 indices
    {
        convexityDefects(contour[i], hullsI[i], defects[i]);
    }
}

Document