how can I detect if a view matrix is left-handed or right-handed?

Question

I have a camera view matrix which I received from a GL program. I know that this matrix is right-handed since this is the way GL works, but how can I check whether this matrix is right-handed or left-handed programmatically?

For the projection matrix I check if matrix[3][4] (row major) is positive to see if it's left handed. Is this correct?

Thanks.

EDIT:

I have tried the determinant solution, but unfortunately it is not true (at least according to my experiments):

I have used DX9 math functions to test it (to avoid any possible bugs in my code). I have run the following code:

 D3DXVECTOR3 vEye(0,0,0);
 D3DXVECTOR3 vTarget(6,3,0);
 D3DXVECTOR3 vUp(0,0,1);

 D3DXMATRIX matViewLH;
 D3DXMATRIX matViewRH;

 D3DXMatrixLookAtLH(&matViewLH, &vEye, &vTarget, &vUp);
 D3DXMatrixLookAtRH(&matViewRH, &vEye, &vTarget, &vUp);

 float fLHDet = D3DXMatrixDeterminant(&matViewLH);
 float fRHDet = D3DXMatrixDeterminant(&matViewRH);

And the two determinants were equal (both equal to 0.99999994), and obviously had the same sign.

As for my problem - Since I get both the view matrix and projection matrix, and it's relatively easy for me to test whether the projection matrix is LH or RH - I use this information to identify the coordinate system.

Emergency

Answer 1

You should take the determinant of the matrix. Other things being equal, left-handed and right-handed matrices should have determinants with opposite signs.

It's a little weird because "left-handed" and "right-handed" are pretty much arbitrary conventions. But, matrices that reflect the world like a mirror have negative determinants, so multiplying by such a matrix will change the sign of the determinant.

---

Edit (ref. updated OQ): I suspect the difference between the D3DX*LH() and D3DX*RH() functions is that they support 2 different conventions.

If so, you should be aware that "left-handed" and "right-handed" is not an attribute of each individual matrix, but of how the matrices generated by these sets of functions fit together. If you stick with the *LH() functions, everything should work out properly; similarly with the *RH() functions -- but if you mix them, you will likely get unexpected results.

In any case, neither of these conventions is relevant to GL, which has its own set of conventions, not fully compatible with either of D3D's. For example, GL's convention for Z clipping is different from D3D's; this means that the projection matrix to generate the same view will necessarily differ between the systems.

So, the short answer to your question is "probably neither".

Abroach

Not really arbitrary. If positive X is to the right, and positive Y is up - then left handed means positive Z forward - away from the viewer, and right handed means positive Z towards the viewer. And I'm not sure if you answer solves my problem: I'm not trying to compare 2 matrices with the same parameters, I'm trying to determine for a specific view matrix which way its positive Z going. – Emergency

Create a known right-handed matrix and compare determinants with your matrix. If they differ you have a left handed matrix. Otherwise it is right handed. – Desert

@Asaf, you are correct. However, my point was that handedness is independent of the GL system. GL doesn't know anything about handedness; it's a consequence of whatever convention the user chooses for his application. – Abroach

-1 - Thanks for the answer, but unfortunately I tested it and it does not seem to be correct (see my question post for details) – Emergency

Hmm, I checked out the D3DX reference, and it's not clear exactly what they mean by "left-handed". The difference between the MatrixLookAtLH() and MatrixLookAtRH() matrices should be a 180 degree rotation about the resulting yaxis -- the matrix itself should always have (as you discovered) a determinant of 1. Perhaps the problem is that you're trying to apply a D3DX convention to a GL system, where it doesn't apply? – Abroach

D3DXMatrixLookAtLH uses the actual forward vector from eye to at, while D3DXMatrixLookAtRH uses a "backwards" forward vector (from at to eye). – Intercalate

Answer 2

Just bound this same task (just for detecting mirror-transformed objects), found an easy solution, sharing it. If the matrix has a 3x3 part containing 3 perpendicular axes, then you can just make a cross product of axes 1 & 2, then do a scalar product of this result and the remaining 3rd axle. Check if it's negative or positive - you can decide about right/left handedness / object mirrored or not. If I understood the question right...

Joaniejoann

Answer 3

Unfortunately, a view matrix itself doesn't carry information about the handedness of the coordinate system. A (correctly constructed) view matrix always fulfills the order of the axis relationships:

z = cross(x, y)
x = cross(y, z)
y = cross(z, x)

This order applies to both left- and right-handed coordinate systems and respective view matrices. And that is why approaches based on the determinant or a concatenation of cross and dot/scalar products are of no help here (both ultimately amount to the same thing). The view matrix always represents an orientation, but not a handedness-dependent mirroring or anything like that.

But luckily you still have the projection matrix, which you have already used for solving your problem. And it is true that you can check matrix[3][4], but only for perspective projection matrices. For the sake of completeness, I would like to note that the general approach, which would also be valid for orthogonal projection matrices, would probably be to check matrix[3][3] to see whether it contains a positive (left-handed) or negative (right-handed) value. And this is where the circle closes, because a right-handed coordinate system has a mirrored Z-axis compared to a left-handed one.

Pyrogallol