Interpreting the Reprojection Error from camera calibration

Question

I'm kind of confused about the Reprojection Error during camera calibration. I understood that the Reprojection Error describes the differences between the detected point and the world point. I even found out that a value <1 is "good".

But what does it tell? I mean e.g. the Reprojection Error is 2: That means the distance is 2px, so far so good. But what does it mean in reference to the calibration? Is a calibration required? or will this value get adjusted by the calibration process to 0?

To be more general: What does the value causes/tells us?

Anhydrous

Answer 1

From Mathworks:

Reprojection errors provide a qualitative measure of accuracy. A reprojection error is the distance between a pattern keypoint detected in a calibration image, and a corresponding world point projected into the same image. The showReprojectionErrors function provides a useful visualization of the average reprojection error in each calibration image. If the overall mean reprojection error is too high, consider excluding the images with the highest error and recalibrating.

Think about it like this. Let's say you're calibrating a face. Your model assumes each of the keypoints of your face to be a certain proportion apart. Of course, each photo will have slightly different different proportional differences than your model. And your model takes like 100 of them, and averages them out to determine the "average" proportions of the face. But what if 10 of those pictures were at weird angles, or were distorted somehow. They are outliers, and they might be throwing off your model. Perhaps it's better to exclude them from your model calculation so you can get a more normalized model of what a normal face looks like. You can tell what is whacked out by looking at the reprojection errors.

Loney

Ok, understood this. Thats like the MatLab manual is saying as well. They suggest to remove thoses images with a large reprojection error value for calibration. But what if all images have a reprojection error <1? I mean what benefit do I have (exept that I following the suggestion)? – Anhydrous

Then your test picture matches your model very well, and if you're happy with the amount of error, then you don't need to do anything. It's just a diagnostic tool for you to see where to improve if you need to improve. – Loney

ah ok I think that was the thing that was missing in my understanding. So is the assumption right if my reprojection error is i.g. <1 then there is just a little distortion (which is ok) but if my reprojection error is i.g. 5 then an calibration (-> undistortion) is necessary? – Anhydrous

Yeah. Let's say you have 999 pictures of you, and 1 picture of someone wearing a highly distorted Halloween mask. Your model takes an average of all those 1000 faces. The reprojection error of your model (the average) should be very close (aka very small) for the 999 pictures of you, but way off for the Halloween mask (aka the reprojection error will be high). Thus, you might want to take the Halloween mask out of your training set. – Loney

Ok thx for that, I understand the issue about "sorting bad images out for training data", but I have a different scenario (I don’t wat to train data). I want an undistortioned image in order to get "straight" lines on the actual image. So if ALL images have an reprojection error > 1 I have to run the camera calibration process (while if all my images have an reprojection error < 1 I don’t have to do it)? – Anhydrous

Let's use numbers as an analogy. If your data set is [100, 101, 101, 99, 99] then your average is 100. Your errors are [0, 1, 1, 1, 1] respectively--pretty close. But let's say your data is [0, 0, 0, 1000, 1000]. Your average is 400. But your errors are [400, 400, 400, 600, 600] respectively. Then it's probably best to cluster into [0, 0, 0], [1000, 1000] and run different models on both sets. But what if you had [0, 0, 0, 0, 1000]? Maybe 1000 is an outlier. Your projection error bar graph will look really high for the 1000. So if you take that 1000 out, your average will be a better fit. – Loney

thanks for all your patience. And thanks for this analogy ;) I think I get it more and more. I believe I understand whats causing the errors. But the errors are only detected when I use the calibration pattern (i.g. chessboard). How does the reprojection error helps me after the camera calibration process? Especially when I use during the calibration process only the data which are causing a "good" error rate (removing Outlinie for instance)? "Real" images can be bad as well without noticing due do no calibration pattern is present on this images. – Anhydrous

This is where math goes from being a science to an art. You just have to develop a feel for it, and decide as the data scientist/machine learning researcher when you should change your models, and how. You'd probably use this to identify outliers, or to identify the performance of your model on the cross validation set. Either way, the reprojection error is just a tool. There is no one way to use it. Sort of like your speedometer in your car. It tells you how fast you're going, but you have to determine as a human whether or not to go over the speed limit, and by how much. – Loney

So to sum up: The reprojection error during camera calibration does not provide any information about how "good" or "bad" the camera calibration is. Nor how big is the optical distortion of the camera is? It's more how good was the picture of my dataset compared to the other from the same dataset? – Anhydrous

Yes. It's the error between the calibration image and the test image for each key point. If I helped, would you mind upvoting and marking as answer? – Loney

Answer 2

The "reprojection error" is a term used for slightly different quantities depending on source. Including:

• A single 2d error vector between measured and projected point
• RMS of all lengths of errors
• Mean of all lengths of errors

OpenCV's cv::calibrateCamera() returns the RMS of all lengths. Two RPE values obtained with different setups are not directly comparable. Pixel size, image noise, lens distortion and camera model play a role.

The RPE will never be exactly 0px due to image noise and lens blur, which cap detection precision and the camera model (e.g. Brown-Conrady) which cannot model the complex lens/sensor-combination 100% accurately.

A value of 2.0px after calibration is somewhat high for a standard quality camera with medium to long focal length. Some sources suggest RPE < 1.0px to generally be acceptable but this depends highly on the type of camera, the quality of the calibration process and the accuracy requirements you have for the application.

You must interpret the value carefully (overfitting) and a low value is not equal to an accurate calibration. You can use it to spot issues in calibration and to compare two calibrations of the same camera system.

An in-depth article on the topic can be found here: https://calib.io/blogs/knowledge-base/understanding-reprojection-errors

Segregationist