Matrix calculations for gpu skinning

Question

I'm trying to do skeletal animation in OpenGL using Assimp as my model import library.

What exactly do I need to the with the bones' offsetMatrix variable? What do I need to multiply it by?

Celeriac

How familiar are you with skinning? It is not enough to use a single matrix for your entire mesh. Have you walked through any tutorial (e.g. this)? – Scintillator

I actually passed the first issue I posted here. I found the bugs that needed fix. I used matrix.inverse() instead of transpose() to convert from row-major to column-major. Stupid me... Anyway, I edited my post and I still have one question left. – Celeriac

And by the way, the answer in this thread seems to be misleading? The author of the answer clearly says that "...If you apply the identity matrix to every joint in the model then what you will get is your model in the bind pose". I tried using identity matrices (instead of the regular node hierarchy matrix multiplications) and I get a weird looking model. – Celeriac

Please post your draw code including the vertex shader. – Scintillator

pastebin.com/CUUurQ2J . Note that I didn't even start with the skinning process (for now, I only pass the vertices to the shader, not even UVs), as I'm still struggling with understanding the basic stuff like the aiBone's mOffsetMatrix data member. (they call it inverse bind pose). Edit: I thought you only asked for the shader code, let me add my draw code too, just a sec... – Celeriac

Here's the whole code: pastebin.com/8aLAGLHe . (class Mesh is also defined in node.h for simplicity) – Celeriac

It does not make sense to discuss skinning without the actual skinning. If mOffsetMatrix is the inverse bind bose matrix, you need this to calculate the actual bone transformation for animated meshes. You will also need the transformation from an animation (probably boneTransform = animTransform * mOffsetMatrix). Those are all per-bone matrices and you need to upload all of them to the GPU for skinning. If there is no animation, boneTransform reduces to the identity matrix. – Scintillator

I still don't seem to grasp the idea behind the inverse bind pose matrix. Can you comment about my second comment here? (regarding the "misleading" answer in the other thread). If a bone (represented by a node, in Assimp) has an "inverse bind pose" matrix, shouldn't it be equal to inverse of node.toRoot matrix? (as the toRoot matrix sets the node's vertices to the model's bind pose) – Celeriac

It probably should. However, I haven't worked with Assimp yet and I don't know if they do anything differently. – Scintillator

I just checked that with debugging and my statement is incorrect. :( – Celeriac

Answer 1

Let's take for instance this code, which I used to animate characters in a game I worked. I used Assimp too, to load bone information and I read myself the OGL tutorial already pointed out by Nico.

glm::mat4 getParentTransform()
{
    if (this->parent)
        return parent->nodeTransform;
    else 
        return glm::mat4(1.0f);
}

void updateSkeleton(Bone* bone = NULL)
{ 
    bone->nodeTransform =  bone->getParentTransform() // This retrieve the transformation one level above in the tree
    * bone->transform //bone->transform is the assimp matrix assimp_node->mTransformation
    * bone->localTransform;  //this is your T * R matrix

    bone->finalTransform = inverseGlobal // which is scene->mRootNode->mTransformation from assimp
        * bone->nodeTransform  //defined above
        * bone->boneOffset;  //which is ai_mesh->mBones[i]->mOffsetMatrix


    for (int i = 0; i < bone->children.size(); i++) {
        updateSkeleton (&bone->children[i]);
    }
}

Essentially the GlobalTransform as it is referred in the tutorial Skeletal Animation with Assimp or properly the transform of the root node scene->mRootNode->mTransformation is the transformation from local space to global space. To give you an example, when in a 3D modeler (let's pick Blender for instance) you create your mesh or you load your character, it is usually positioned (by default) at the origin of the Cartesian plane and its rotation is set to the identity quaternion.

However you can translate/rotate your mesh/character from the origin (0,0,0) to somewhere else and have in a single scene even multiple meshes with different positions. When you load them, especially if you do skeletal animation, it is mandatory to translate them back in local space (i.e. back at the origin 0,0,0 ) and this is the reason why you have to multiply everything by the InverseGlobal (which brings back your mesh to local space).

After that you need to multiply it by the node transform which is the multiplication of the parentTransform (the transformation one level up in the tree, this is the overall transform) the transform (formerly the assimp_node->mTransformation which is just the transformation of the bone relative to the node's parent) and your local transformation (any T * R) you want to apply to do: forward kinematic, inverse kinematic or key-frame interpolation.

Eventually there is the boneOffset (ai_mesh->mBones[i]->mOffsetMatrix) that transforms from mesh space to bone space in bind pose as stated in the documentation.

Here there is a link to GitHub if you want to look at the whole code for my Skeleton class.

Hope it helps.

Achilles

Was struggling with this for weeks. Posted questions everywhere on the web. Your concept of having nodeTransform did the trick. Thanks you! – Celeriac

Glad it helped :-) I should write an article about it I think! – Achilles

Hey man, I'm diving into this subject again as I want to build a better system of skeletal animation. I want to be sure about something, please answer the next question: Every aiMesh refers to some bone/s with an offset matrix. Can a specific bone have different offsetMatrix in different aiMeshes? – Celeriac

Hey Pilpel, did you ever find an answer to your last question? I suspect the answer is no but I can't find a definitive answer. I assume you're trying to decouple meshes and skeletons to allow for re-use of one skeleton with multiple meshes? – Tobytobye

Answer 2

The offset matrix defines the transform (translation, scale, rotation) that transforms the vertex in mesh space, and converts it to "bone" space. As an example consider the following vertex and a bone with the following properties;

Vertex Position<0, 1, 2>

Bone Position<10, 2, 4>
Bone Rotation<0,0,0,1>  // Note - no rotation
Bone Scale<1, 1, 1>

If we multiply a vertex by the offset Matrix in this case we would get a vertex position of <-10, -1, 2>.

How do we use this? You have two options on how to use this matrix which is down to how we store the vertex data in the vertex buffers. The options are;

1) Store the mesh vertices in mesh space 2) Store the mesh vertices in bone space

In the case of #1, we would take the offsetMatrix and apply it to the vertices that are influenced by the bone as we build the vertex buffer. And then when we animate the mesh, we later apply the animated matrix for that bone.

In the case of #2, we would use the offsetMatrix in combination with the animation matrix for that bone when transforming the vertices stored in the vertex buffer. So it would be something like (Note: you may have to switch the matrix concatenations around here);

anim_vertex = (offset_matrix * anim_matrix) * mesh_vertex

Does this help?

Onwards

I'm curious how you would store vertices in bone-space. Would you store all vertices multiple times for every influenced bone? – Scintillator

Yup, that's pretty much how I would do it. In this case, I would perform the bone transforms on the CPU or on the GPU using stream-out and a uniform block holding the bone matrices. I try to avoid doing this sort of thing in the triangle render pipe via vertex shaders, to keep shader complexity to a minimum and to simplify the overall render system design. – Onwards

It "could" be but keep in mind that for a single vertex, you have to perform an additional matrix multiply but I guess you could cache the inverse offset * anim offset matrices. But for four influences, you're still multiplying the vertex multiple times regardless if you store the verts in bone or mesh space.The extra cost really comes from the input bandwidth for local bone positions, normals and tangents. – Onwards

you could cache the inverse offset * anim offset matrices That's exactly the proper way to do it. I've never seen anyone upload both matrices. The only thing you'd be saving is doing that matrix multiplication on the CPU whenever you update the skeleton, at the expense of making the vertex shader more heavy. – Princess

Answer 3

As I already assumed, the mOffsetMatrix is the inverse bind pose matrix. This tutorial states the correct transformations that you need for linear blend skinning:

You first need to evaluate your animation state. This will give you a system transform from animated bone space to world space for every bone (GlobalTransformation in the tutorial). The mOffsetMatrix is the system transform from world space to bind pose bone space. Therefore, what you do for skinning is the following (assuming that a specific vertex is influenced by a single bone): Transform the vertex to bone space with mOffsetMatrix. Now assume an animated bone and transform the intermediate result back from animated bone space to world space. So:

boneMatrix[i] = animationMatrix[i] * mOffsetMatrix[i]

If the vertex is influenced by multiple bones, LBS simply averages the results. That's where the weights come into play. Skinning is usually implemented in a vertex shader:

vec4 result = vec4(0);
for each influencing bone i
    result += weight[i] * boneMatrix[i] * vertexPos;

Usually, the maximum number of influencing bones is fixed and you can unroll the for loop.

The tutorial uses an additional m_GlobalInverseTransform for the boneMatrix. However, I have no clue why they do that. Basically, this undoes the overall transformation of the entire scene. Probably it is used to center the model in the view.

Scintillator

Couldn't understand that line: No assume an animated bone and transform.... Can you rewrite it? Also, correct me if I'm wrong, but since the vertices for each mesh are stored in their node space, boneMatrix should be (order is right to left): finalBoneMatrix = Inverse(offsetMatrix) * animationMatrix * offsetMatrix * meshNode.toRoot;..? A textual order: node space to root space (aka world space), world space to bone space, bone space to animated (bone) space, animated space to root space. Did I get anything wrong? – Celeriac

There was a w missing. You have the vertex position in bone space and then continue transforming to world space by assuming an animated bone. There are two hierarchies: the skeleton hierarchy and the mesh hierarchy. If the mesh hierarchy contains additional transforms, you need to add them as you wrote. But that's rather unusual for skinned meshes. The first Inverse(offsetMatrix) is wrong. This would transform a vertex from bone space to world space. But you are already in world space after the transformation with animationMatrix. – Scintillator

But animationMatrix is basically just a T/R matrix that I get from the animation structure, based on a timestamp. Why would it transform the vertices to world space? – Celeriac

Because that's what the animation Matrix does. It can be thought of as a model transform that positions a bone in world space. E.g. the animation matrix of an upper arm bone would contain a translation that moves the bone from the origin to the shoulder position and it might contain an additional rotation. – Scintillator

"but since the vertices for each mesh are stored in their node space" They are not. They are stored in object space. – Princess