How to transform a vec3 by a mat4?

Question

I'm translating a bit of code into javascript. I'm trying to replicate the functionality of

 D3DXVECTOR3* D3DXVec3TransformCoord(
 __inout  D3DXVECTOR3 *pOut,
 __in     const D3DXVECTOR3 *pV,
 __in     const D3DXMATRIX *pM
 );

This applies Mat4 transformation to the Vec3 (which would lead to an invalid mat4). How do I project the 4d matrix into the 3d matrix and perform the operation? In what order would you do the math? We can assume that W=1, as in the following post:

DirectX Math function confusion

Constitutionalism

Are you really asking how to transform a matrix by a vector? – Colincolinson

Look here: gamedev.net/topic/11906-d3dxvec3transformcoord – Sunstone

Well, between built in c++ and D3DXVec3Transform and D3DXVec3TransformCoord, I just want to be certain about what exact math they perform. You never know if its doing a little more or a little less than you expect based on a name. That link helps. Now I know where to find the internals. – Constitutionalism

related question: https://mcmap.net/q/867572/-how-can-i-transform-a-glm-vec3-by-a-glm-mat4/6908282 – Slangy

Answer 1

The documentation for this function says "Transforms a 3D vector by a given matrix, projecting the result back into w = 1".

In other words the function extends the vector [x,y,z] to [x,y,z,1] and then performs the multiplication, truncating the result to three elements after the operation.

EDIT: According to this thread the operation is xA, i.e multiplication with a row vector on the left, which is equivalent to the homogeneous transformation: xM(0) + yM(1) + zM(2) + M(3), where M(i) is the i'th column of the matrix.

Adumbral

Is this the same as normal c++ Vec3 * Mat4? I see that operator getting used, but I wonder what it's actually doing. – Constitutionalism

Mathematically Vec3 * Mat4 is not defined, the vector length has to match the number of rows in the matrix. If you take the w=1 "projection" into account then you should get the same result as a Vec4 * Mat4 routine. – Adumbral

Answer 2

Other than building it yourself you can also use an existing npm package called gl-matrix which has a transformMat4 function in the vec3.js module of the package.

OR

you can use the below function to do the transformation yourself

function transformMat4AG(out, a, m) {
    var x = a[0],
        y = a[1],
        z = a[2];
    var w = m[3] * x + m[7] * y + m[11] * z + m[15];
    w = w || 1.0;
    out[0] = (m[0] * x + m[4] * y + m[8] * z + m[12]) / w;
    out[1] = (m[1] * x + m[5] * y + m[9] * z + m[13]) / w;
    out[2] = (m[2] * x + m[6] * y + m[10] * z + m[14]) / w;
    return out;
}

Slangy