I have a camera view matrix which I received from a GL program. I know that this matrix is right-handed since this is the way GL works, but how can I check whether this matrix is right-handed or left-handed programmatically?
For the projection matrix I check if matrix[3][4] (row major) is positive to see if it's left handed. Is this correct?
Thanks.
EDIT:
I have tried the determinant solution, but unfortunately it is not true (at least according to my experiments):
I have used DX9 math functions to test it (to avoid any possible bugs in my code). I have run the following code:
D3DXVECTOR3 vEye(0,0,0);
D3DXVECTOR3 vTarget(6,3,0);
D3DXVECTOR3 vUp(0,0,1);
D3DXMATRIX matViewLH;
D3DXMATRIX matViewRH;
D3DXMatrixLookAtLH(&matViewLH, &vEye, &vTarget, &vUp);
D3DXMatrixLookAtRH(&matViewRH, &vEye, &vTarget, &vUp);
float fLHDet = D3DXMatrixDeterminant(&matViewLH);
float fRHDet = D3DXMatrixDeterminant(&matViewRH);
And the two determinants were equal (both equal to 0.99999994), and obviously had the same sign.
As for my problem - Since I get both the view matrix and projection matrix, and it's relatively easy for me to test whether the projection matrix is LH or RH - I use this information to identify the coordinate system.