OpenCV RotatedRect with specified angle

Question

I have the situation that I have a small binary image that has one shape, around which I want to find the best fitting rotated rectangle (not bounding rectangle). I know that there is cv::minAreaRect() that you apply on the result found by cv::findContours(), but this has delivered poor results in my case, because the data is noisy (coming from MS Kinect, see example picture where rotation changes due to the input data (contour) being slightly different). What I did instead was to calculate the principal axis using PCA on my binary image (which is less sensitive to noise), which yields angle "a", and now I want to create a RotatedRect around my shape, given the angle of the principal axis, a).

I have an illustration, made with my superb Paint skills!

So then my question is: do you guys have code snippets or concrete suggestions to solve this? I'm afraid that I have to do many Bresenham iterations, hoping that there is a clever approach.

Btw, for those who are not too familiar with the RotatedRect data structure of openCV: it is defined by height, width, angle, and center point, assuming that center point is actually, well, in the center of the rectangle.

Cheers!

Burnell

Wouldn't you rotate the shape by a, then fit the rectangle? – Bipropellant

Rotating it is not an option. It can not be assumed that the shape is actually in the center of the image. It could also be in the lower left corner. My current idea of solving this problem is to do <width of picture> Bresenham traversals, maybe intelligent enough to not scan those areas I already know have been scanned, trying to maximize the width and height of the rotated rect. – Burnell

What do you mean by "this has delivered poor results" ? Can you share a sample image you're working with? – Whaley

It's poor because of noise sensitivity, I'll edit the post above with a picture. – Burnell

It turned out that rotation is indeed helpful. Since I get the mean vector of my binary image as a side product of the PCA, I rotated the contour points around that point (with angle = -a) and determined the simple bounding box. The results are "OK", although they're not perfectly aligned. Still, width and height of the found rectangle are 100% correct, and I'm thinking about fixing the smaller errors by doing the Bresenham traversal with lines that are orthogonal to my angle='a' vector, to find e.g. the true top and left border. – Burnell

Answer 1

OK, my solution: Approach:

1. PCA, gives the angle and a first approximation for the rotatedRect's center
2. Get the contour of the binary shape, rotate it into upright position, get min/max of X and Y coordinates to get the width and height of the bounding rect
3. Subtract half the width (height) from maximum X (Y) to get the center point in the "upright space"

4. Rotate this center point back by the inverse rotation matrix

   cv::RotatedRect Utilities::getBoundingRectPCA( cv::Mat& binaryImg ) {
   cv::RotatedRect result;
   
   //1. convert to matrix that contains point coordinates as column vectors
   int count = cv::countNonZero(binaryImg);
   if (count == 0) {
       std::cout << "Utilities::getBoundingRectPCA() encountered 0 pixels in binary image!" << std::endl;
       return cv::RotatedRect();
   }
   
   cv::Mat data(2, count, CV_32FC1);
   int dataColumnIndex = 0;
   for (int row = 0; row < binaryImg.rows; row++) {
       for (int col = 0; col < binaryImg.cols; col++) {
           if (binaryImg.at<unsigned char>(row, col) != 0) {
               data.at<float>(0, dataColumnIndex) = (float) col; //x coordinate
               data.at<float>(1, dataColumnIndex) = (float) (binaryImg.rows - row); //y coordinate, such that y axis goes up
               ++dataColumnIndex;
           }
       }
   }
   
   //2. perform PCA
   const int maxComponents = 1;
   cv::PCA pca(data, cv::Mat() /*mean*/, CV_PCA_DATA_AS_COL, maxComponents);
   //result is contained in pca.eigenvectors (as row vectors)
   //std::cout << pca.eigenvectors << std::endl;
   
   //3. get angle of principal axis
   float dx = pca.eigenvectors.at<float>(0, 0);
   float dy = pca.eigenvectors.at<float>(0, 1);
   float angle = atan2f(dy, dx)  / (float)CV_PI*180.0f;
   
   //find the bounding rectangle with the given angle, by rotating the contour around the mean so that it is up-right
   //easily finding the bounding box then
   cv::Point2f center(pca.mean.at<float>(0,0), binaryImg.rows - pca.mean.at<float>(1,0));
   cv::Mat rotationMatrix = cv::getRotationMatrix2D(center, -angle, 1);
   cv::Mat rotationMatrixInverse = cv::getRotationMatrix2D(center, angle, 1);
   
   std::vector<std::vector<cv::Point> > contours;
   cv::findContours(binaryImg, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
   if (contours.size() != 1) {
       std::cout << "Warning: found " << contours.size() << " contours in binaryImg (expected one)" << std::endl;
       return result;
   }
   
   //turn vector of points into matrix (with points as column vectors, with a 3rd row full of 1's, i.e. points are converted to extended coords)
   cv::Mat contourMat(3, contours[0].size(), CV_64FC1);
   double* row0 = contourMat.ptr<double>(0);
   double* row1 = contourMat.ptr<double>(1);
   double* row2 = contourMat.ptr<double>(2);
   for (int i = 0; i < (int) contours[0].size(); i++) {
       row0[i] = (double) (contours[0])[i].x;
       row1[i] = (double) (contours[0])[i].y;
       row2[i] = 1;
   }
   
   cv::Mat uprightContour = rotationMatrix*contourMat;
   
   //get min/max in order to determine width and height
   double minX, minY, maxX, maxY;
   cv::minMaxLoc(cv::Mat(uprightContour, cv::Rect(0, 0, contours[0].size(), 1)), &minX, &maxX); //get minimum/maximum of first row
   cv::minMaxLoc(cv::Mat(uprightContour, cv::Rect(0, 1, contours[0].size(), 1)), &minY, &maxY); //get minimum/maximum of second row
   
   int minXi = cvFloor(minX);
   int minYi = cvFloor(minY);
   int maxXi = cvCeil(maxX);
   int maxYi = cvCeil(maxY);
   
   //fill result
   result.angle = angle;
   result.size.width = (float) (maxXi - minXi);
   result.size.height = (float) (maxYi - minYi);
   
   //Find the correct center:
   cv::Mat correctCenterUpright(3, 1, CV_64FC1);
   correctCenterUpright.at<double>(0, 0) = maxX - result.size.width/2;
   correctCenterUpright.at<double>(1,0) = maxY - result.size.height/2;
   correctCenterUpright.at<double>(2,0) = 1;
   cv::Mat correctCenterMat = rotationMatrixInverse*correctCenterUpright;
   cv::Point correctCenter = cv::Point(cvRound(correctCenterMat.at<double>(0,0)), cvRound(correctCenterMat.at<double>(1,0)));
   
   result.center = correctCenter;
   
   return result;
   

   }

Burnell

Answer 2

If understand the problem correctly, you're saying the method of using findContours and minAreaRectsuffers from jitter/wobbling due to the noisy input data. PCA is not more robust against this noise, so I don't see why you think finding the orientation of the pattern this way won't be as bad as your current code.

If you need temporal smoothness a commonly used and simple solution is to use a filter, even a very simple filter like an alpha-beta filter probably gives you the smoothness you want. Say at frame n you estimate the parameters of the rotated rectangle A, and in frame n+1 you have the rectangle with the estimated parameters B. Instead of drawing the rectangle with B you find C which is between A and B, and then draw a rectangle with C in frame n+1.

Whaley

First, smoothing angles is not trivial because -179° is very similar to 179°, and a smoothed value is meaningless. Second, I do believe that PCA is a lot more robust against small outliers. After all, the covariance matrix should change only little because it is calculated on ALL points inside the contour. When two or three noisy pixels come into play additionally, this won't change the matrix much. But it may change the minAreaRect() rectangle by several degrees. – Burnell

Answer 3

Here's another approach (just a guess)

Wikipedia page on Principal Component Analysis says:

PCA can be thought of as fitting an n-dimensional ellipsoid to the data ...

And as your data is 2D, you can use the cv::fitEllipse function to fit an ellipse to your data and use the coordinates of the generated RotatedRect to calculate the angle. This gives better results as compared to cv::minAreaRect.

Flair