Emgu (OpenCV for C#) - building a disparity map using cvStereoRectify

Question

I've been using the great Emgu C# wrapper for OpenCV to collect images from a home built stereo rig. Two webcams bolted down to a piece of wood, 35cm apart to hopefully let me produce depth maps in the 10-20m range. I've set them up to be as parallel as I can make it (about 89.3 degrees from a triangulation test).

I'm trying to make a disparity map from these and although the process works as code the results are very random. By this I mean each time I attempt to run the stereo rectify I get very different results and the image is often so badly warped there is hardly anything to see on screen.

As I understand it the way to do this is as follows:

---

1) Print off a chessboard pattern (e.g 6 by 8 inner corners)- stick onto something flat.

2) Take a set of about 10 photos from camera 1 holding the chess board in full view but different positions.

3) use CameraCalibration.FindChessboardCorners to find the inner corners (6 by 8)

4) use img.FindCornerSubPix() to refine these corner locations to subpixel level

5) use CameraCalibration.CalibrateCamera() to calculate the intrinsic camera details, and save this as an XML file

6) Repeat above for camera 2.

7) Now you have the internal camera distortion information you can take a stereo photo pair and use CameraCalibration.StereoCalibrate() with the intrinsic data previously calculated to work out the extrinsic information (offsets and rotations between camera 1 and 2).

8) Use CvInvoke.cvStereoRectify() and CvInvoke.cvInitUndistortRectifyMap() and then CvInvoke.cvRemap() to build an output image which should be lined up in Y so you can run one of the stereo correspondence tests.

I found you need to be using Emgu 2.1 ver 806 to get the cvStereoRectify to work without an Access Violation error.

---

I guess my questions are:

A) Is my process correct? I've been doing the camera intrinsic calibration as a separate process because as the cameras are 35cm apart it's not easy to get the chessboard in view of both of them in the office and move it around much.. as it soon goes off the side of one of the camera views. I figured that as the values are intrinsic these are related to the camera and therefore should transfer to the stereo procedure OK. Is this correct?

It seems that the intrinsic values are getting altered during the cvStereoRectify process, and are being made very different.

eg. Distortion values from first stage = 0.22,-1.2,0.01,-0.01,2.6 after cvStereoRectify the values were changed to= 10,-489,-0.03,-0.09,13208

I'm no expert but the first set seem to be more like what I'd seen from other people's comments, and the second set seem pretty way out!

B) Is there any way to stop the intrinsic + distortion values being updated during cvStereoRectify?

C) Does this seem right for the intrinsic values (937,0,290,0,932,249,0,0,1)?

Thanks very much for any tips... I've been stuck on this for a while now... and I'm really not sure which part of the process is throwing up the errors. Any tips or suggestions would be very welcome...

Miceli

I've just discovered that everything seems to work better if you do collect the image set from L and R cameras at the same time, then solve all of the object model points (ie chessboard corner coords) and the two sets of image corner coords (ie left and right camera sub pixel coords). – Miceli

Also it seems to do a better job when the chessboard is fairly close to the camera... my problem is that this doesn't give me much room to work as my cameras are so far apart. Therefore I've just made a larger chessboard (200%) and hold it further away.. giving me the accuracy in corner matching and also more room to move it around. Seems to help... still working on it! – Miceli

@timemirror: Have you checked out the Matlab calibration toolbox(www.vision.caltech.edu/bouguetj/calib_doc/) or Danail Stoyonov's Camera Calibration Tools (ubimon.doc.ic.ac.uk/dvs/m581.html) both are excellent tools for calibration. By the way what were your findings with the emguCV library, did you get satisfactory results with it? – Armandoarmature

You could verify your findings from the emguCV library with these tools, you can also find many links to other calibration toolboxes at this link vision.caltech.edu/bouguetj/calib_doc/htmls/links.html – Armandoarmature

@Kevin Boyd - thanks for the links, and comments. I don't have access to Matlab unfortunately. I found Emgu to be very good.I found that there were two main issues in my camera calibration.. a) I had to calibrate both of the cameras in the stereo rig at the same time b) the rig used two cheap webcams which had manual focus, and one of the focus rings was slightly loose so when the cameras were moved after calibration the focus would change ever so slightly, but enough to through out my previous calibrations. In the end I switched to a higher quality video cameras, and Emgu worked flawlessly. – Miceli

I'm trying to do close to the same thing, and had one question about the Emgu CameraCalibration.CalibrateCamera() function. Did you (or anyone else here) have to do any pre-processing on the IntrinsicCameraParameters arguments? Like starting it out to be assigned with a certain value? Also, the function takes an ExtrinsicCameraParameters[] instead of (my expected) ExtrinsicCameraParameters object as an 'out' argument. Could someone provide insight on how to deal with those two arguments? Or tell me if there is any dealing to do with them? I'm getting a 'NaN' value as the return. Peace. – Gaskins

Answer 1

I haven't used EMGU in a couple of years, but I was using it to calibrate a camera and a Kinect.

I found that there can be some confusion with the ordering of the rectangle corners if the rectangle is completely horizontal - I never really got to the bottom of why, and this just seemed to work for me. Try setting the board up so that the rectangle is at a roll of 45 degrees (i.e. from the front you would be look at more of a diamond shape, if that makes any sense).

My very basic understanding is that this should make it easy for both cameras to ensure that they pick the same corner as the first corner to start counting from.

I can't guarantee that this will fix your problem - it could just be that I had something deeply wrong in my code and this was why it failed to produce regular results when the chessboard was flat. Then again, it won't take you long to test and might just work.

Pearliepearline