How is the reprojection error calculated in Matlab's triangulate function? Sadly, the documentation gives no mathematical formula

Question

How is the reprojection error calculated in Matlab's triangulate function?

Sadly, the documentation gives no mathematical formula.

It only says: The vector contains the average reprojection error for each M world point.

What is the procedure/Matlab uses when calculating this error?

I searched SOF but found nothing on this IMHO important question.

UPDATE: How can they use this error to filter out bad matches here : http://se.mathworks.com/help/vision/examples/sparse-3-d-reconstruction-from-two-views.html

Portecochere

Answer 1

AFAIK The reprojection error is calculated always in the same way (in the field of computer vision in general).

The reprojection is (as the name says) the error between the reprojected point in the camera and the original point.

So from 2 (or more) points in the camera you triangulate and get 3D points in the world system. Due to errors in the calibration of the cameras, the point will not be 100% accurate. What you do is take the result 3D point (P) and with the camera calibration parameters project it in the cameras again, obtaining new points (\hat{p}) near the original ones (p).

Then you calculate the euclidean distance between the original point and the "reprojected" one.

In case you want to know a bit more about the method used by Matlab, I'll enhance the bibliography they use giving you also the page number:

Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman (p312). Cambridge University Press, 2003.

But basically it is a least squares minimization, that has no geometrical interpretation.

Lineation

Yes, but what I don't understand is how they use this error to filter out bad matches : se.mathworks.com/help/vision/examples/… – Portecochere

If I am not mistaken, bad matches or bad correspondences are not filtered out with the re-projection error per say, in the mathworks example there could be an implicit RANSAC stage where it removes outliers, as either part of the feature extraction or feature point tracker (its using optical flow?; opencv has RANSAC in the Lucas-Kande function). Further, under the hood, the "triangulate" function could be using a non-linear optimization approach to refine 3D point estimates, as suppose to the linear approaches such as 8 point algorithm where it is very prone to noise and explode (numerically). – Thunderpeal

@Thunderpeal Does it matter how any of that is performed? The reprojection error will still be the reprojection error. – Lineation

Clarify what you mean by "any of that", what particular thing are we talking about? If we are talking about bad matches, you should get rid of them as soon as possible (i.e feature detection, extraction stage) before using them to calculate the reprojection error, bad points will affect reprojection errors, and give you bad estimates. – Thunderpeal

@Thunderpeal I just mean that yes, id doesnt matter what method you use, the reporjection error is the reprojection error, and that is what OP asked. It has nothing to do with feature matching or anything, its just the error on the calibration side. – Lineation

@AnderBiguri I was responding to jhegedus first comment about his misunderstanding with regards to using reprojection error to filter out bad matches, it doesn't. – Thunderpeal

@Thunderpeal you are right. But you may come a bit late to the party, this is about a year and a half ago! – Lineation

Your point being? StackOverflow is a Q&A site, not a forum. Many questions on SO are old but still applicable, especially this question. The author made a bad assumption that he could use reprojection error to filter bad matches, my simple response was you can't. Whats the problem here? Please enlighten me. – Thunderpeal

@Thunderpeal wow chill, there is absolutely no problem, nor I tried to say there was one. No need for that tone. – Lineation

Answer 2

Above mentioned answers interpret re-projection error in a simplistic way as an actual reprojection in the camera. In more general sense, this error reflects the distance between a noisy image point and the point estimated from the model. One can imagine a tangental plane to some surface (model) in n-dimensional space where the noisy point is projected (hence it lands on the plane, not on the model!). n is not obligatory = 2 since a notion of a "point" can be generalized to, for example, concatenation of coordinates of two corresponding points for Homography.

It is important to understand that reprojection error is not a final answer. Overall_error^2 = reprojection_error^2 + estimation_error^2. The latter is the distance between estimation reprojected and true point on the model. More on this can be found in chapter 5 of Hatrtley andd Zisserman book Multiple View Geometry. They prove that reprojeciton error has a theoretical limit 0.6*sigma (for Homography estimation), where sigma is noise standard deviation.

Eyelid

Answer 3

You can find an explanation of reprojection errors in the context of camera calibration in the Camera Calibrator tutorial:

The reprojection errors returned by the triangulate function are essentially the same concept.

The way to use the reprojection errors to discard bad matches is shown in this example:

[points3D, reprojErrors] = triangulate(matchedPoints1, matchedPoints2, ...
    cameraMatrix1, cameraMatrix2);

% Eliminate noisy points
validIdx = reprojErrors < 1;
points3D = points3D(validIdx, :);

This code excludes all 3D points for which the reprojection error was more than a pixel. You can also use validIdx to eliminate the corresponding 2D matches.

Rooks

I see, but how can this error be used to discard bad matches: se.mathworks.com/help/vision/examples/… – Portecochere

@jhegedus, I have created a chat room to talk about 3D reconstruction with matlab. Please join if you have more questions. I am also curious to know more about what you are doing. chat.stackoverflow.com/rooms/73265/… – Rooks

@jhegedus, or can you drop me a note via my Matlab Central profile? mathworks.com/matlabcentral/answers/contributors/… – Rooks

Hi Dima, thank you for the chatroom, I should join, what time is good for you tomorrow? – Portecochere

I understand that it is used like this (to exclude bad matches) but what I don't understand is why can it be used like this. – Portecochere

I am just about to pass out... my brain is barely functional anymore today... tomorrow would be good anytime, during daytime (utc +2, Helsinki time) – Portecochere

In that case, just send me a note through my Matlab Central profile, whenever you feel better. Then we can continue this conversation over email. – Rooks

Thank you very much Dima, it seems that this was just a short, two week pilot-project to see how easy/difficult 3D reconstruction would be. It seems that it is not at all trivial, so this project is now on hold but perhaps I return to it later. It depends on my employer. Keep in touch and thanks again ! – Portecochere

Answer 4

They filter out bad matches by removing relevant indexes which have large re-projection errors.

This means that points that have large re-projection errors are the outliers.

Lychnis