Problems with opencv stereoRectifyUncalibrated

Question

I've been trying to rectify and build the disparity mappping for a pair of images using OpenCV stereoRectifyUncalibrated, but I'm not getting very good results. My code is:

template<class T>
T convertNumber(string& number)
{
    istringstream ss(number);
    T t;
    ss >> t;
    return t;
}

void readPoints(vector<Point2f>& points, string filename)
{
    fstream filest(filename.c_str(), ios::in);
    string line;

    assert(filest != NULL);

    getline(filest, line);
    do{
        int posEsp = line.find_first_of(' ');
        string posX = line.substr(0, posEsp);
        string posY = line.substr(posEsp+1, line.size() - posEsp);

        float X = convertNumber<float>(posX);
        float Y = convertNumber<float>(posY);

        Point2f pnt = Point2f(X, Y);
        points.push_back(pnt);
        getline(filest, line);
    }while(!filest.eof());

    filest.close();
}

void drawKeypointSequence(Mat lFrame, Mat rFrame, vector<KeyPoint>& lKeyp, vector<KeyPoint>& rKeyp)
{
    namedWindow("prevFrame", WINDOW_AUTOSIZE);
    namedWindow("currFrame", WINDOW_AUTOSIZE);
    moveWindow("prevFrame", 0, 300);
    moveWindow("currFrame", 650, 300);
    Mat rFrameAux;
    rFrame.copyTo(rFrameAux);
    Mat lFrameAux;
    lFrame.copyTo(lFrameAux);

    int size = rKeyp.size();
    for(int i=0; i<size; i++)
    {
        vector<KeyPoint> drawRightKeyp;
        vector<KeyPoint> drawleftKeyp;

        drawRightKeyp.push_back(rKeyp[i]);
        drawleftKeyp.push_back(lKeyp[i]);

        cout << rKeyp[i].pt << " <<<>>> " << lKeyp[i].pt << endl;

        drawKeypoints(rFrameAux, drawRightKeyp, rFrameAux, Scalar::all(255), DrawMatchesFlags::DRAW_OVER_OUTIMG);
        drawKeypoints(lFrameAux, drawleftKeyp, lFrameAux, Scalar::all(255), DrawMatchesFlags::DRAW_OVER_OUTIMG);

        imshow("currFrame", rFrameAux);
        imshow("prevFrame", lFrameAux);
        waitKey(0);
    }
    imwrite("RightKeypFrame.jpg", rFrameAux);
    imwrite("LeftKeypFrame.jpg", lFrameAux);
}
int main(int argc, char* argv[])
{
    StereoBM stereo(StereoBM::BASIC_PRESET, 16*5, 21);
    double ndisp = 16*4;
    assert(argc == 5);
    string rightImgFilename(argv[1]);       // Right image (current frame)
    string leftImgFilename(argv[2]);        // Left image (previous frame)
    string rightPointsFilename(argv[3]);    // Right image points file
    string leftPointsFilename(argv[4]);     // Left image points file

    Mat rightFrame = imread(rightImgFilename.c_str(), 0);
    Mat leftFrame = imread(leftImgFilename.c_str(), 0);

    vector<Point2f> rightPoints;
    vector<Point2f> leftPoints;

    vector<KeyPoint> rightKeyp;
    vector<KeyPoint> leftKeyp;

    readPoints(rightPoints, rightPointsFilename);
    readPoints(leftPoints, leftPointsFilename);
    assert(rightPoints.size() == leftPoints.size());

    KeyPoint::convert(rightPoints, rightKeyp);
    KeyPoint::convert(leftPoints, leftKeyp);

    // Desenha os keypoints sequencialmente, de forma a testar a consistência do matching
    drawKeypointSequence(leftFrame, rightFrame, leftKeyp, rightKeyp);

    Mat fundMatrix = findFundamentalMat(leftPoints, rightPoints, CV_FM_8POINT);
    Mat homRight;
    Mat homLeft;
    Mat disp16 = Mat(rightFrame.rows, leftFrame.cols, CV_16S);
    Mat disp8 = Mat(rightFrame.rows, leftFrame.cols, CV_8UC1);
    stereoRectifyUncalibrated(leftPoints, rightPoints, fundMatrix, rightFrame.size(), homLeft, homRight);

    warpPerspective(rightFrame, rightFrame, homRight, rightFrame.size());
    warpPerspective(leftFrame, leftFrame, homLeft, leftFrame.size());


    namedWindow("currFrame", WINDOW_AUTOSIZE);
    namedWindow("prevFrame", WINDOW_AUTOSIZE);
    moveWindow("currFrame", 650, 300);
    moveWindow("prevFrame", 0, 300);
    imshow("currFrame", rightFrame);
    imshow("prevFrame", leftFrame);

    imwrite("RectfRight.jpg", rightFrame);
    imwrite("RectfLeft.jpg", leftFrame);

    waitKey(0);

    stereo(rightFrame, leftFrame, disp16, CV_16S);

    disp16.convertTo(disp8, CV_8UC1, 255/ndisp);
    FileStorage file("disp_map.xml", FileStorage::WRITE);
    file << "disparity" << disp8;
    file.release();
    imshow("disparity", disp8);
    imwrite("disparity.jpg", disp8);
    moveWindow("disparity", 0, 0);

    waitKey(0);
}

drawKeyPoint sequence is the way I visually check the consistency of the points I have for both images. By drawing each of their keypoints in sequence, I can be sure that keypoint i on image A is keypoint i on image B.

I've also tried playing with the ndisp parameter, but it didn't help much.

I tried it for the following pair of images:

LeftImage

RightImage

got the following rectified pair:

RectifiedLeft

RectifiedRight

and finally, the following disparity map

DisparityMap

Which, as you can see, is quite bad. I've also tried the same pair of images with the following stereoRectifyUncalibrated example: http://programmingexamples.net/wiki/OpenCV/WishList/StereoRectifyUncalibrated and the SBM_Sample.cpp from opencv tutorial code samples to build the disparity map, and got a very similar result.

I'm using opencv 2.4

Thanks in advance!

Kinny

I am facing very similar problems. Did you find a solution? – Machellemachete

Answer 1

Besides possible calibration problems, your images clearly lack some texture for the stereo block matching to work. This algorithm will see many ambiguities and too large disparities on flat (non-tetxured) parts.

Note however that the keypoints seem to match well, so even if the rectification output seems weird it is probably correct.

You can test your code against standard images from the Middlebury stereo page for sanity checks.

Kilovoltampere

Answer 2

I would suggest to do a stereo calibration using the chessboard, or take multiple pictures with a chess board and use stereocalibrate.cpp on your computer. I am saying that because you are using stereorectifyuncalibrated, While the algorithm does not need to know the intrinsic parameters of the cameras, it heavily depends on the epipolar geometry. Therefore, if the camera lenses have a significant distortion, it would be better to correct it before computing the fundamental matrix and calling this function. For example, distortion coefficients can be estimated for each head of stereo camera separately by using calibrateCamera(). Then, the images can be corrected using undistort() , or just the point coordinates can be corrected with undistortPoints().

Repressive