What is the Project Tango lens distortion model?

Question

The Project Tango C API documentation says that the TANGO_CALIBRATION_POLYNOMIAL_3_PARAMETERS lens distortion is modeled as:

x_corr_px = x_px (1 + k1 * r2 + k2 * r4 + k3 * r6) y_corr_px = y_px (1 + k1 * r2 + k2 * r4 + k3 * r6)

That is, the undistorted coordinates are a power series function of the distorted coordinates. There is another definition in the Java API, but that description isn't detailed enough to tell which direction the function maps.

I've had a lot of trouble getting things to register properly, and I suspect that the mapping may actually go in the opposite direction, i.e. the distorted coordinates are a power series of the undistorted coordinates. If the camera calibration was produced using OpenCV, then the cause of the problem may be that the OpenCV documentation contradicts itself. The easiest description to find and understand is the OpenCV camera calibration tutorial, which does agree with the Project Tango docs:

But on the other hand, the OpenCV API documentation specifies that the mapping goes the other way:

My experiments with OpenCV show that its API documentation appears correct and the tutorial is wrong. A positive k1 (with all other distortion parameters set to zero) means pincushion distortion, and a negative k1 means barrel distortion. This matches what Wikipedia says about the Brown-Conrady model and will be opposite from the Tsai model. Note that distortion can be modeled either way depending on what makes the math more convenient. I opened a bug against OpenCV for this mismatch.

So my question: Is the Project Tango lens distortion model the same as the one implemented in OpenCV (documentation notwithstanding)?

Here's an image I captured from the color camera (slight pincushioning is visible):

And here's the camera calibration reported by the Tango service:

distortion = {double[5]@3402}
[0] = 0.23019999265670776
[1] = -0.6723999977111816
[2] = 0.6520439982414246
[3] = 0.0
[4] = 0.0
calibrationType = 3
cx = 638.603
cy = 354.906
fx = 1043.08
fy = 1043.1
cameraId = 0
height = 720
width = 1280

Here's how to undistort with OpenCV in python:

>>> import cv2
>>> src = cv2.imread('tango00042.png')
>>> d = numpy.array([0.2302, -0.6724, 0, 0, 0.652044])
>>> m = numpy.array([[1043.08, 0, 638.603], [0, 1043.1, 354.906], [0, 0, 1]])
>>> h,w = src.shape[:2]
>>> mDst, roi = cv2.getOptimalNewCameraMatrix(m, d, (w,h), 1, (w,h))
>>> dst = cv2.undistort(src, m, d, None, mDst)
>>> cv2.imwrite('foo.png', dst)

And that produces this, which is maybe a bit overcorrected at the top edge but much better than my attempts with the reverse model:

Conductive

Answer 1

The Tango C-API Docs state that (x_corr_px, y_corr_px) is the "corrected output position". This corrected output position needs to then be scaled by focal length and offset by center of projection to correspond to a distorted pixel coordinates.

So, to project a point onto an image, you would have to:

1. Transform the 3D point so that it is in the frame of the camera
2. Convert the point into normalized image coordinates (x, y)
3. Calculate r2, r4, r6 for the normalized image coordinates (r2 = x*x + y*y)

4. Compute (x_corr_px, y_corr_px) based on the mentioned equations:

   x_corr_px = x (1 + k1 * r2 + k2 * r4 + k3 * r6)
   y_corr_px = y (1 + k1 * r2 + k2 * r4 + k3 * r6)
   

5. Compute distorted coordinates

   x_dist_px = x_corr_px * fx + cx
   y_dist_px = y_corr_px * fy + cy
   

6. Draw (x_dist_px, y_dist_px) on the original, distorted image buffer.

This also means that the corrected coordinates are the normalized coordinates scaled by a power series of the normalized image coordinates' magnitude. (this is the opposite of what the question suggests)

Looking at the implementation of cvProjectPoints2 in OpenCV (see [opencv]/modules/calib3d/src/calibration.cpp), the "Poly3" distortion in OpenCV is being applied the same direction as in Tango. All 3 versions (Tango Docs, OpenCV Tutorials, OpenCV API) are consistent and correct.

Good luck, and hopefully this helps!

(Update: Taking a closer look at a the code, it looks like the corrected coordinates and distorted coordinates are not the same. I've removed the incorrect parts of my response, and the remaining parts of this answer are still correct.)

Radborne

Thanks! That answers my question. However, I strongly disagree that there is no problem in the docs. Saying "corrected" == "distorted" sounds Orwellian :-). Mapping from a (single canonical) calibrated coordinate space to an (infinite number of) uncalibrated coordinate space is not correction. – Conductive

Also please note that tango_client_api.h documents TANGO_CALIBRATION_POLYNOMIAL_3_PARAMETERS as "Tsai's K1, K2, K3 model". The Tsai model maps distorted coordinates to undistorted coordinates, which is not what you're saying above. – Conductive

I took another look at Tsai's paper. Looking at the equation just after 5b on page 327, you're right, in that the Tsai polynomial distortion maps from corrected (distorted) coordinates to undistorted (pinhole model) coordinates. Sorry about the doc discrepancy in the Tango API. I'll see what I can do to get this corrected! – Radborne

"I'll see what I can do to get this corrected!" You mean, you'll see what you can do to get this distorted? ;-) – Conductive

Well, played @Conductive :-). I'll take the hint and see what I can do to simplify the distorted/corrected nomenclature in the docs. – Radborne

@Conductive - In an attempt to disambiguate distorted & correted, I took a deeper look into the code, and (x_corr_px, y_corr_px) is an intermediate coordinate system that is normalized(ish). These corrdinates still need to be scaled by focal length and center of projection to become distorted coordinates. – Radborne

Three releases later, tango_client_api.h still wrongly claims to be using the Tsai model. – Conductive

Answer 2

Maybe it's not the right place to post, but I really want to share the readable version of code used in OpenCV to actually correct the distortion.

I'm sure that I'm not the only one who needs x_corrected and y_corrected and fails to find an easy and understandable formula.

I've rewritten the essential part of cv2.undistortPoints in Python and you may notice that the correction is performed iteratively. This is important, because the solution for polynom of 9-th power does not exist and all we can do is to apply its the reveresed version several times to get the numerical solution.

def myUndistortPoint((x0, y0), CM, DC):
    [[k1, k2, p1, p2, k3, k4, k5, k6]] = DC
    fx, _, cx = CM[0]
    _, fy, cy = CM[1]

    x = x_src = (x0 - cx) / fx
    y = y_src = (y0 - cy) / fy

    for _ in range(5):
        r2 = x**2 + y**2
        r4 = r2**2
        r6 = r2 * r4

        rad_dist = (1 + k4*r2 + k5*r4 + k6*r6) / (1 + k1*r2 + k2*r4 + k3*r6)
        tang_dist_x = 2*p1 * x*y + p2*(r2 + 2*x**2)
        tang_dist_y = 2*p2 * x*y + p1*(r2 + 2*y**2)

        x = (x_src - tang_dist_x) * rad_dist
        y = (y_src - tang_dist_y) * rad_dist

    x = x * fx + cx
    y = y * fy + cy

    return x, y

To speed up, you can use only three iterations, on most cameras this will give enough precision to fit the pixels.

Sift