Comparing a saved movement with other movement with Kinect

Question

I need to develop an application where a user (physiotherapist) will perform a movement in front of the Kinect, I'll write the data movement in the database and then the patient will try to imitate this motion. The system will calculate the similarity between the movement recorded and executed.

My first idea is, during recording (each 5 second, by example), to store the position (x, y, z) of the points and then compare them in the execution time(by patient).

I know that this approach is too simple, because I imagine that in people of different sizes the skeleton is recognized differently, so the comparison is not reliable.

My question is about the best way to compare a saved motion with a movement executed (on the fly).

Garble

A friend suggested me to "normalize" the sizes of the physiotherapist and the patient but i think this is not a good ideia. – Garble

$%*# I am doing this too – Abstracted

I just use ellipse and map them to a canvas and multiply there position using the algorithem: whereItIs * (whereYouWantItToBe / whereItIs) – Abstracted

Outlaw Lemur, can you explai this in details? – Garble

If you still need help just ask – Abstracted

Meet me in the Kinect Developement Chatroom – Abstracted

Answer 1

I have done this, where a doctors frame is projected onto the patients frame, but with the whole skeleton this doesn't work so well because of different bone heights :/. The code can be found here. It is in beta 2 code, the more current version can be found here, although it is not currently working perfectly

As for comparing, do something like this

for (int i = 0; i < patientList.Count; i++)
{
    int diff = (int)Math.Abs(patientList[i] - doctorList[i]);

    if (diff < 100) //or whatever number you want
    {
         Debug.WriteLine("Good Job");
    }
}

I have abandoned the idea of a whole figure because of the bone heights mentioned by Fixus, so my current program looks some thing like:

EDIT

This is the concept of camparing two movements with kinect and calculate a similarity between the two movements I explain in depth.

Suppose I have the following 2 points, point A (0, 0, 0) and point B (1, 1, 1). Now I want to find the difference from point A to B, so I would subtract all of the X, Y, and Z numbers, so the difference is 1 X 1 Y 1 Z. That is the simple stuff. Now to implement it. The code I have written above, I would implement like this.

//get patient hand coordinates
double patienthandX = Canvas.GetLeft(patienthand);
double patienthandY = Canvas.GetTop(patienthand);

//get doctor hand coordinates
double doctorhandX = Canvas.GetLeft(doctorhand);
double doctorhandY = Canvas.GetTop(doctorhand);

//compare difference for each x and y
//take Absolute value so that it is positive
double diffhandX = Math.Abs(patienthandX - doctorhandX);
double diffhandY = Math.Abs(patienthandY - doctorhandY);

Now here comes another issue. The doctor coordinates are always the same, but what if the patient isn't standing where the doctor coordinates were recorded? Now we implement more simple math. Take this simple example. suppose I want point A(8, 2) to move to point B(4, 12). You multiply the x and y's of A to get to B. So I would multiply the X by .5, and the Y by 6. So for Kinect, I would put a element on the patients hip, then compare this to the doctors hip. Then multiply all of the doctor joints by that number to achieve the doctor joints on top of the patients (more or less). For example

double whatToMultiplyX = (double) doctorhipX / patienthipX;
double whatToMultiplyY = (double) doctorhipY / patienthipY;

This is all pretty simple, but bringing it together is the harder part. So far we, 1) Scale the doctor frames on top of the patient frames, 2) Calculate the difference. 3) Compare the difference throughout the entire rep. and 4) Reset for the next rep. This seems simple but it is not. To calculate the entire difference for the rep, do something like this:

//get patient hand coordinates
double patienthandX = Canvas.GetLeft(patienthand);
double patienthandY = Canvas.GetTop(patienthand);

//get doctor hand coordinates
double doctorhandX = Canvas.GetLeft(doctorhand);
double doctorhandY = Canvas.GetTop(doctorhand);

//compare difference for each x and y
//take Absolute value so that it is positive
double diffhandX = Math.Abs(patienthandX - doctorhandX);
double diffhandY = Math.Abs(patienthandY - doctrorhandY);

//+= so that it keeps adding to it.
totaldiffhandX += diffhandX;
totaldiffhandY += diffhandY;

Now we can compare, and say:

if (totaldiffhandX < 1000 && totaldiffhandY < 1000) //keep numbers pretty high since it is an entire rep
{
     //reset difference
     totaldiffhandX = 0;
     totaldiffhandY = 0;

     //tell the patient good job
     Debug.WriteLine("Good Job");
}

This is pretty easy, but keep in mind you must do this for every single joint's x and y. Otherwise it will not work. Hope this Helps.

Abstracted

Answer 2

First of all remember that people are diffrent. Every person has diffrent height, width, weight, diffrent bones length etc etc

You`re code probably will never work cause of this.

Secondly you need to think more geometrically. Don`t think about points only, think with vectors, their directions. Each movement is movent of some vectors in some directions. Then the proportion. You need to configure application for each user.

You have some pattern. The patter is your physiotherapist. You need to remember not only his movements but also his body. Arm length, leg length, distances etc. Each user that will be using your app also need to me mesured. Having all this data you can compare movement by scaling sizes and comparing directions of movent

Of course remember that there are some very simple moves like for example. They can be recognized by simple mathematic by checking actual position of the hand and checking direction of the movement. You need for this 3 control points and you`re at home :)

Gesture recognizing isn`t a simple thing

Emsmus

Interesting, in thorical field it work, but i am about to implement the code, i need some concrete direction. :) – Garble

@Garble dont get me wrong but this is lot of coding, implementing math so dont count that Ill write it for you. You need to understand that recognizing gesture (when you want to recognize advanced move like whole body) isnt a simple thing and it is hard work. If you need help with theory or with code that won`t work for you feel free to ask. Remember that this is not a one day job if you want to do it correct – Emsmus

@Emsmus So true, I have pumped in ~250 hours in this project so far – Abstracted

I agree, and i dont asking for code, this is MY work i excited with it and i will do it. When i said concrete, i mean something like: "Yeah man, o already achieve this goal in my last project and i used the following strategy: bla-bla-bla. :) – Garble

-1 as this only describes the problems, and the very general way to fix it. But not in extreme depth. – Abstracted