How to detect the line of sight of a person using kinect?

Question

Currently I am working on a project using Kinect which requires me to know the where the person is looking at that time, for which I figured out I need to find the line of sight of that person.

Right now, I can find the head point of the skeleton of the person but can't track the eye movement.

if (body.TrackingState == SkeletonTrackingState.Tracked)
{
    Joint joint = body.Joints[JointType.Head];
    SkeletonPoint skeletonPoint = joint.Position;

    // 2D coordinates in pixels
    System.Drawing.Point point = new System.Drawing.Point();

    if (_mode == CameraMode.Color)
    {
        // Skeleton-to-Color mapping
        ColorImagePoint colorPoint = _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(skeletonPoint, ColorImageFormat.RgbResolution640x480Fps30);

        point.X = colorPoint.X;
        point.Y = colorPoint.Y;
        //Console.WriteLine(" X == " + point.X + " Y == " + point.Y);
        X = (int)Math.Floor(point.X + 0.5);
        Y = (int)Math.Floor(point.Y + 0.5);
    }

    // DRAWING...
    Ellipse ellipse = new Ellipse
    {
        Fill = System.Windows.Media.Brushes.LightBlue,
        Width = 20,
        Height = 20
    };

     Canvas.SetLeft(ellipse, point.X - ellipse.Width / 2);
     Canvas.SetTop(ellipse, point.Y - ellipse.Height / 2);

     canvas.Children.Add(ellipse);
 }

Here point.X and point.Y are the head points of the skeleton.

Gauntry

I haven't messed with the Kinect SDK in forever but isn't it possible to detect the rotation of the head? – Cesya

@JohnOdom how do you think to use rotation of the head in my problem....cant get your idea..though i can figure out changes in head position using skeleton coordinates – Gauntry

I was thinking of if you were able to keep track of the head's rotation (0 degree for when facing the kinect) you can figure out the "line of sight" by checking how much it rotated. Ex: If the head rotated +45 degrees on the yaw (up/down Z axis) then the user is looking to the left. It may sound like overkill but it's all I can offer without looking into the kinect sdk. – Cesya

@JohnOdom 2 things 1. Rotation of the person around the yaw will not change his head points 2. your method neglects the movement of the eye without the movement of head. – Gauntry

I know I'm neglecting the eye movement, I wasn't sure if the kinect was advance enough to read eye movements. For the rotation I meant the head's rotation, not the body. I also wasn't sure if the Kinect could keep track of the body part's rotation individually. – Cesya

Not a Kinect developer, but based on what I could find, 2.0 has facial tracking (well, they've had it since 1.5 it appears), but as far as eyes are concerned, the extent is "open or closed". (See "Developer" section, "Does Kinect windows Windows SDK offer facial recognition capabilities?" sub-section) – Keirakeiser

See if this SO question helps. – Hultin

Which version of the SDK are you using? – Plicate

@VitoShadow sdk v1.8 – Gauntry

Answer 1

Have you looked at the FaceBasics Sample Project?

I believe you want to use the FaceFrameSource / FaceFrameReader (note: not HDFace). You'll be able to get the face orientation as a Quarternion (and the sample project translates that to Pitch/Yaw/Roll).

Combining that with the 3D location of the head from the skeleton, I think you should be able to create an approximate line of sight.

The How-to Videos cover Face including some information on Orientation (5th video, skip in about 18:20 - your specific question is asked at 21:49).

EDIT: Rough proof of concept showing modifications made to FaceBasics Sample Project - added to ~line 565, right after the face info is drawn (I also needed to change the scope of pitch/yaw/roll defined a few lines above and set their default values to 0). This creates a circle for a head, and a yellow line looking at an approximate gaze location.

Joint HeadJoint = this.bodies[faceIndex].Joints[JointType.Head];
ColorSpacePoint colorPoint = this.coordinateMapper.MapCameraPointToColorSpace(HeadJoint.Position);
Point HeadPoint = new Point(colorPoint.X, colorPoint.Y);
Point GazePoint = new Point(HeadPoint.X - Math.Sin((double)yaw * 0.0175) * 600, HeadPoint.Y - Math.Sin((double)pitch * 0.0175) * 600);
drawingContext.DrawLine(new Pen(System.Windows.Media.Brushes.Yellow, 5), HeadPoint, GazePoint);
drawingContext.DrawEllipse(System.Windows.Media.Brushes.LightBlue, null, HeadPoint, 70, 70);

EDIT 2: Just saw your new comment saying that you are using SDK v1.8 - my answer is going off of v2.0, and I can't speak for how things would be different with the older SDK/Sensor.

Vacation

yes I have seen those videos, and modifying the facebasics code for line of sight seem to me a very difficult task as it contained different modules which were interdependent on each other...thanks – Gauntry

I think you're going to have a very hard time if you want to track face orientation but without using the FaceFrameSource. – Vacation

just got the kinect v2 and adapter...will be implementing your suggestions now...will let you know about further updates – Gauntry