Kinect sideways skeleton tracking

Question

Currently I am using the Microsoft Kinect for measuring angles between joints. Most measurements are working correctly. Whenever a person is sitting sideways (on a chair) the Kinect won't track the skeleton accurate. To illustrate my problem I've added 3 pictures of the Kinect depthview.

As you can see 2 out of 3 measurements work "correctly". Whenever I lift my leg, the Kinect stops skeleton tracking correctly. Does anyone have a solution to this problem, or is this just a limitation of the Kinect?

Thanks.

Update 1: The JointTrackingState-Enumeration on these tracked joints shown at screenshot 2 are marked as Inferred, however the depth view is tracking my full body.

Update 2: At screenshot 2 I'm trying to track my front leg, highlighted as green. I know the other leg isn't tracked but this does not matter, I guess.

Update 3: The following code selects a skeleton:

private Skeleton StickySkeleton(Skeleton[] skeletons)
{
    if (skeletons.Count<Skeleton>(skeleton => skeleton.TrackingId == _trackedSkeletonId) <= 0)
    {
        _trackedSkeletonId = -1;
        _skeleton = null;
    }

    if (_trackedSkeletonId == -1)
    {
        Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton => skeleton.TrackingState == SkeletonTrackingState.Tracked);

        if (foundSkeleton != null)
        {
            _trackedSkeletonId = foundSkeleton.TrackingId;
            return foundSkeleton;
        }
    }

    return _skeleton;
}

Whenever a skeleton is tracked that data will be used for drawing the joint-points and calculating the angle between joints.

Update 4: I've tested sitting on a 'block', far more simplified than a chair. Unfortunately the Kinect still acts the same.

Below 2 screenshots:

Fee

I have had many problems, when i sit on something or took something in hands. So i think that this is limitation of the kinect. – Mycenae

However my full body is highlighted in screenshot 1 and 2. So I can't see why the Kinect isn't able to track my leg. The JointTrackingState value on these joints are marked as Inferred while the depth view is tracking my leg.. – Fee

Do you mean your back leg isnt seen as well? – Placido

Screenshot 2 doesn't show my backleg tracked but this does not cause any problems with tracking the other leg I presume? In situations where the backleg is being tracked, the same errors occur. – Fee

@MarkvanVelthoven imo the problem is that Kinect has no estimation to the floor. On screen 1 and 3 youre touching the floor, kinect is using this to mesure everything. On screen 2 youre "in the air" does not know what to do. That is my imo – Biliary

So what about games like: Kinect Sports. They're jumping and therefor "in the air" and the Kinect still knows what to do. Any suggestions? – Fee

Have you tried sitting on something simpler then the chair you're sitting on in the pictures? Perhaps something like a block? – Contradistinction

I've found and tried something similar to a block. The following two screenshots show the behaviour of the Kinect: Sitting on a block 1 and Setting on a block 2. Currently I'm not touching the ground but the kinect still acts the same way as I would touch the ground on a chair.. – Fee

which sample you are using? did you test it KinectExplorer sample? you have to show some code – Plectrum

I've tried KinectExplorer sample containted inside the Microsoft Kinect SDK. The following screenshots show the behaviour of the Kinect using the KinectExplorer sample: Screenshot 1 and Screenshot2. As you can see, same result. See my main post for code, thanks! – Fee

On screen 2 did you start being tracked then lifted your legs? Or were you never tracked to begin with in any of the photos? – Placido

I bet this is because in picture 5, where kinect would think your knee is, there is another protrusion(your leg), so I thinks you have 1 leg and a weird limb growing out of your knee – Placido

This is a limitation in the Kinect SDK (Skeletal tracking) however you are free to place in your own skeletal tracking algorithm from the depth image. I am not going to place this as an answer but hope this excels how you think about the issue. This is how I would do it: – Kial

Note: You have to be sure to use kinect for windows to enable sitting mode and have it actually use sitting mode. the x-box sitting mode does nothing. – Beckham

Answer 1

As Renaud Dumont stated, I would do something with JointTrackingState. Since your using knees, I used the variables leftknee and rightknee which are Joints to do it. Here's code, You might use JointType.FootRight and JointType.FootLeft and the Hip types, but I'll leave that up to you.

static Skeleton first = new Skeleton();

Joint leftknee = first.Joints[JointType.KneeLeft];
Joint rightknee = first.Joints[JointType.KneeRight];

if ((leftknee.TrackingState == JointTrackingState.Inferred ||
                leftknee.TrackingState == JointTrackingState.Tracked) && 
                (rightknee.TrackingState == JointTrackingState.Tracked ||
                rightknee.TrackingState == JointTrackingState.Inferred))
            {

            }

Or alternately, if you just wanted one knee to be tracked at a time, or both, you could do this:

if ((leftknee.TrackingState == JointTrackingState.Inferred ||
                leftknee.TrackingState == JointTrackingState.Tracked) && 
                (rightknee.TrackingState == JointTrackingState.Tracked ||
                rightknee.TrackingState == JointTrackingState.Inferred))
            {

            }

            else if (leftknee.TrackingState == JointTrackingState.Inferred ||
                    leftknee.TrackingState == JointTrackingState.Tracked)
            {

            }

            else if (rightknee.TrackingState == JointTrackingState.Inferred ||
                    rightknee.TrackingState == JointTrackingState.Tracked)
            {

            }

FYI the reason the Skeleton is first is static is because then it can be used in making the joints

 static Skeleton first;

Opposed to

 Skeleton first;

Edit 1

I've come to the conclusion this is extremely difficult to do, I think the above method will work, but I just wanted to include what I'm working on in case you might be able to find some way to make it work. Anyways Here's the code I was working on which is another class which is just another SkeletalTrackingState I was trying to make an Inferred enum in it. But unfortunately enum are impossible to inherit. If you find something to this effect that works, I will respect you as a superior programmer to me forever ;). Without further ado: The .dll I was trying to make:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Kinect;

namespace IsInferred
{
public abstract class SkeletonInferred : Skeleton
{
    public bool inferred;
    static Skeleton first1 = new Skeleton();
    Joint handright;
    Joint handleft;
    Joint footright;
    Joint footleft;
    Joint ankleleft;
    Joint ankleright;
    Joint elbowleft;
    Joint elbowright;
    Joint head;
    Joint hipcenter;
    Joint hipleft;
    Joint hipright;
    Joint shouldercenter;
    Joint shoulderleft;
    Joint shoulderright;
    Joint kneeleft;
    Joint kneeright;
    Joint spine;
    Joint wristleft;
    Joint wristright;

    public SkeletonInferred(bool inferred)
    {

    }

    public enum Inferred
    {
        NotTracked = SkeletonTrackingState.NotTracked,
        PositionOnly = SkeletonTrackingState.PositionOnly,
        Tracked = SkeletonTrackingState.Tracked,
        Inferred = 3,
    }

    private void IsInferred(object sender, AllFramesReadyEventArgs e)
    {
        handright = first1.Joints[JointType.HandRight];
        handleft = first1.Joints[JointType.HandLeft];
        footright = first1.Joints[JointType.FootRight];
        footleft = first1.Joints[JointType.FootLeft];
        ankleleft = first1.Joints[JointType.AnkleLeft];
        ankleright = first1.Joints[JointType.AnkleRight];
        elbowleft = first1.Joints[JointType.ElbowLeft];
        elbowright = first1.Joints[JointType.ElbowRight];
        head = first1.Joints[JointType.Head];
        hipcenter = first1.Joints[JointType.HipCenter];
        hipleft = first1.Joints[JointType.HipLeft];
        hipright = first1.Joints[JointType.HipRight];
        shouldercenter = first1.Joints[JointType.ShoulderCenter];
        shoulderleft = first1.Joints[JointType.ShoulderLeft];
        shoulderright = first1.Joints[JointType.ShoulderRight];
        kneeleft = first1.Joints[JointType.KneeLeft];
        kneeright = first1.Joints[JointType.KneeRight];
        spine = first1.Joints[JointType.Spine];
        wristleft = first1.Joints[JointType.WristLeft];
        wristright = first1.Joints[JointType.WristRight];

        if (handleft.TrackingState == JointTrackingState.Inferred &
            handright.TrackingState == JointTrackingState.Inferred &
            head.TrackingState == JointTrackingState.Inferred &
            footleft.TrackingState == JointTrackingState.Inferred &
            footright.TrackingState == JointTrackingState.Inferred &
            ankleleft.TrackingState == JointTrackingState.Inferred &
            ankleright.TrackingState == JointTrackingState.Inferred &
            elbowleft.TrackingState == JointTrackingState.Inferred &
            elbowright.TrackingState == JointTrackingState.Inferred &
            hipcenter.TrackingState == JointTrackingState.Inferred &
            hipleft.TrackingState == JointTrackingState.Inferred &
            hipright.TrackingState == JointTrackingState.Inferred &
            shouldercenter.TrackingState == JointTrackingState.Inferred &
            shoulderleft.TrackingState == JointTrackingState.Inferred &
            shoulderright.TrackingState == JointTrackingState.Inferred &
            kneeleft.TrackingState == JointTrackingState.Inferred &
            kneeright.TrackingState == JointTrackingState.Inferred &
            spine.TrackingState == JointTrackingState.Inferred &
            wristleft.TrackingState == JointTrackingState.Inferred &
            wristright.TrackingState == JointTrackingState.Inferred)
        {
            inferred = true;
        }
      }
    }
  }

The code in your project (compiler error)

    using IsInferred;

    static bool Inferred = false;
    SkeletonInferred inferred = new SkeletonInferred(Inferred);
    static Skeleton first1 = new Skeleton();

    Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton =>  skeleton.TrackingState == SkeletonTrackingState.Inferred);

Good luck, I hope this helps you get going in the right direction or helps you at all!

My Code

Here's my code that you asked for. Yes it is from the Skeletal Tracking Fundamentals, but this code was here and I didn't want to start a new project with most of the same stuff. Enjoy!

Code

     // (c) Copyright Microsoft Corporation.
     // This source is subject to the Microsoft Public License (Ms-PL).
    // Please see http://go.microsoft.com/fwlink/?LinkID=131993 for details.
    // All other rights reserved.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Coding4Fun.Kinect.Wpf; 

namespace SkeletalTracking
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
    }

    bool closing = false;
    const int skeletonCount = 6; 
    Skeleton[] allSkeletons = new Skeleton[skeletonCount];

    private void Window_Loaded(object sender, RoutedEventArgs e)
    {
        kinectSensorChooser1.KinectSensorChanged += new  DependencyPropertyChangedEventHandler(kinectSensorChooser1_KinectSensorChanged);

    }

    void kinectSensorChooser1_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e)
    {
        KinectSensor old = (KinectSensor)e.OldValue;

        StopKinect(old);

        KinectSensor sensor = (KinectSensor)e.NewValue;

        if (sensor == null)
        {
            return;
        }




        var parameters = new TransformSmoothParameters
        {
            Smoothing = 0.3f,
            Correction = 0.0f,
            Prediction = 0.0f,
            JitterRadius = 1.0f,
            MaxDeviationRadius = 0.5f
        };
        sensor.SkeletonStream.Enable(parameters);

        //sensor.SkeletonStream.Enable();

        sensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(sensor_AllFramesReady);
        sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); 
        sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

        try
        {
            sensor.Start();
        }
        catch (System.IO.IOException)
        {
            kinectSensorChooser1.AppConflictOccurred();
        }
    }

    void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
    {
        if (closing)
        {
            return;
        }

        //Get a skeleton
        Skeleton first =  GetFirstSkeleton(e);

        if (first == null)
        {
            return; 
        }



        //set scaled position
        //ScalePosition(headImage, first.Joints[JointType.Head]);
        ScalePosition(leftEllipse, first.Joints[JointType.HandLeft]);
        ScalePosition(rightEllipse, first.Joints[JointType.HandRight]);
        ScalePosition(leftknee, first.Joints[JointType.KneeLeft]);
        ScalePosition(rightknee, first.Joints[JointType.KneeRight]);

        GetCameraPoint(first, e); 

    }

    void GetCameraPoint(Skeleton first, AllFramesReadyEventArgs e)
    {

        using (DepthImageFrame depth = e.OpenDepthImageFrame())
        {
            if (depth == null ||
                kinectSensorChooser1.Kinect == null)
            {
                return;
            }


            //Map a joint location to a point on the depth map
            //head
            DepthImagePoint headDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.Head].Position);
            //left hand
            DepthImagePoint leftDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandLeft].Position);
            //right hand
            DepthImagePoint rightDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandRight].Position);

            DepthImagePoint rightKnee =
                depth.MapFromSkeletonPoint(first.Joints[JointType.KneeRight].Position);

            DepthImagePoint leftKnee =
                depth.MapFromSkeletonPoint(first.Joints[JointType.KneeLeft].Position);


            //Map a depth point to a point on the color image
            //head
            ColorImagePoint headColorPoint =
                depth.MapToColorImagePoint(headDepthPoint.X, headDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //left hand
            ColorImagePoint leftColorPoint =
                depth.MapToColorImagePoint(leftDepthPoint.X, leftDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //right hand
            ColorImagePoint rightColorPoint =
                depth.MapToColorImagePoint(rightDepthPoint.X, rightDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);

            ColorImagePoint leftKneeColorPoint =
                depth.MapToColorImagePoint(leftKnee.X, leftKnee.Y,
                ColorImageFormat.RgbResolution640x480Fps30);

            ColorImagePoint rightKneeColorPoint =
                depth.MapToColorImagePoint(rightKnee.X, rightKnee.Y,
                ColorImageFormat.RgbResolution640x480Fps30);



            //Set location
            CameraPosition(headImage, headColorPoint);
            CameraPosition(leftEllipse, leftColorPoint);
            CameraPosition(rightEllipse, rightColorPoint);


            Joint LEFTKNEE = first.Joints[JointType.KneeLeft];
            Joint RIGHTKNEE = first.Joints[JointType.KneeRight];

            if ((LEFTKNEE.TrackingState == JointTrackingState.Inferred ||
            LEFTKNEE.TrackingState == JointTrackingState.Tracked) &&
            (RIGHTKNEE.TrackingState == JointTrackingState.Tracked ||
            RIGHTKNEE.TrackingState == JointTrackingState.Inferred))
            {
                CameraPosition(rightknee, rightKneeColorPoint);
                CameraPosition(leftknee, leftKneeColorPoint);
            }

            else if (LEFTKNEE.TrackingState == JointTrackingState.Inferred ||
                    LEFTKNEE.TrackingState == JointTrackingState.Tracked)
            {
                CameraPosition(leftknee, leftKneeColorPoint);
            }

            else if (RIGHTKNEE.TrackingState == JointTrackingState.Inferred ||
                    RIGHTKNEE.TrackingState == JointTrackingState.Tracked)
            {
                CameraPosition(rightknee, rightKneeColorPoint);
            }
        }        
    }


    Skeleton GetFirstSkeleton(AllFramesReadyEventArgs e)
    {
        using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame())
        {
            if (skeletonFrameData == null)
            {
                return null; 
            }


            skeletonFrameData.CopySkeletonDataTo(allSkeletons);

            //get the first tracked skeleton
            Skeleton first = (from s in allSkeletons
                                     where s.TrackingState == SkeletonTrackingState.Tracked
                                     select s).FirstOrDefault();

            return first;

        }
    }

    private void StopKinect(KinectSensor sensor)
    {
        if (sensor != null)
        {
            if (sensor.IsRunning)
            {
                //stop sensor 
                sensor.Stop();

                //stop audio if not null
                if (sensor.AudioSource != null)
                {
                    sensor.AudioSource.Stop();
                }


            }
        }
    }

    private void CameraPosition(FrameworkElement element, ColorImagePoint point)
    {
        //Divide by 2 for width and height so point is right in the middle 
        // instead of in top/left corner
        Canvas.SetLeft(element, point.X - element.Width / 2);
        Canvas.SetTop(element, point.Y - element.Height / 2);

    }

    private void ScalePosition(FrameworkElement element, Joint joint)
    {
        //convert the value to X/Y
        //Joint scaledJoint = joint.ScaleTo(1280, 720); 

        //convert & scale (.3 = means 1/3 of joint distance)
        Joint scaledJoint = joint.ScaleTo(1280, 720, .3f, .3f);

        Canvas.SetLeft(element, scaledJoint.Position.X);
        Canvas.SetTop(element, scaledJoint.Position.Y); 

    }


    private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
    {
        closing = true; 
        StopKinect(kinectSensorChooser1.Kinect); 
    }

    private void kinectDepthViewer1_Loaded(object sender, RoutedEventArgs e)
    {

    }

   }
}

XAML

<Window x:Class="SkeletalTracking.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="600" Width="800" Loaded="Window_Loaded" 
    xmlns:my="clr-namespace:Microsoft.Samples.Kinect.WpfViewers;assembly=Microsoft.Samples.Kinect.WpfViewers" 
    Closing="Window_Closing" WindowState="Maximized">       
<Canvas Name="MainCanvas">
    <my:KinectColorViewer Canvas.Left="0" Canvas.Top="0" Width="640" Height="480" Name="kinectColorViewer1" 
                          Kinect="{Binding ElementName=kinectSensorChooser1, Path=Kinect}" />
    <Ellipse Canvas.Left="0" Canvas.Top="0" Height="50" Name="leftEllipse"  Width="50" Fill="#FF4D298D" Opacity="1" Stroke="White" />
    <Ellipse Canvas.Left="100" Canvas.Top="0" Fill="#FF2CACE3" Height="50" Name="rightEllipse" Width="50" Opacity="1" Stroke="White" />
    <my:KinectSensorChooser Canvas.Left="250" Canvas.Top="380" Name="kinectSensorChooser1" Width="328" />
    <Image Canvas.Left="66" Canvas.Top="90" Height="87" Name="headImage" Stretch="Fill" Width="84" Source="/SkeletalTracking;component/c4f-color.png" />
    <Ellipse Canvas.Left="283" Canvas.Top="233" Height="23" Name="leftknee" Stroke="Black" Width="29" />
    <Ellipse Canvas.Left="232" Canvas.Top="233" Height="23" Name="rightknee" Stroke="Black" Width="30" />
</Canvas>

Here's a picture just to show how off Kinect can be sometimes:

Hint: Notice how only my arm is detected and part of the background

Placido

Thanks for your anwser, but could you have a look to the following video. The Kinect is able to track the body sideways, for clarification look to this screenshot. Thanks! This is an official video from Microsoft aka "The Kinect Effect". – Fee

@MarkvanVelthoven Please check edit, I would have commented but I find it is hard to read larger amounts of code in a comments:/ – Placido

Thanks for your constructive comment, sadly enough SkeletonTrackingState does not have a Inferred-status. I've tried PositionOnly-status, this didn't changed anything. Maybe another suggestion? :P – Fee

@MarkvanVelthoven I think he was thinking about the JointTrackingState which is a status assigned to each one of the joints. The PositionOnly status value is related to the Skeleton, and a Skeleton with that status doesn't provide a Joint collection as it is not actually tracked. So you should make sure all the joints are drawn, and not only those with a JointTrackingState.Tracked status. – Dioptrics

@MarkvanVelthoven Ah yes, must have been confusing the two – Placido

@MarkvanVelthoven @RenaudDumont I have decided to make a class that has a Inferred skeleton tracking state, will post edits tomorrow as it is a lengthy bit of code – Placido

@Outlaw Lemur: Thanks a million for your effort and answer. Unfortunately tracking legs is still not accurate. More and more I start to think this is one of the few limitations of the Kinect. P.S. Did you try the code yourself and/or tracking legs? – Fee

@MarkvanVelthoven No, But I will do that – Placido

@MarkvanVelthoven The first method worked perfectally for me, except the knee tracking was a bit off:/, but I thought this worked. Good Luck! – Placido

@Outlaw Lemur Would you be so kind to share your visual studio solution where you've tested te written code above. I might have implemented it differently. Thanks! – Fee

Updated youtube link to Kinect Effect video: youtube.com/watch?v=RN_zu3xVQ5M – Oskar

Answer 2

The kinect is failing to track the skeleton accurately due to limitations in its own SDK. Essentiall every time the device tries to pick up a skeleton, it always assumes that it is front facing. If you save the joints, take a top down view and plot the x and z co-ordinates this is quite easily seen.

Frisco

Answer 3

I am also working with Kinect , you can get the value for the joint angles for joint angles of your knee but the problem that you seem to be facing is due to Kinect itself as it inferred values are not shown . Its better you use the whole skeleton body to track your knee and hopefully you will get all the values for your movement of each joint

Manwell