Detecting a Fist with Leap Motion SDK V2

Question

I want to check if a Hand in a Leap Motion Frame is currently a Fist.

The usually suggested method is to look for hand.grabStrength with a value of 1. The problem is that the value jumps to 1 even with a "Claw-Like" Hand, or anything else with very slightly curled fingers.

Another approach would be to check on each finger if it is extended. But this has a similiar issue, Fingers only count as extended if they are completely straight. So even if i check for all fingers to be not extended, the same issue as above occurs (claw-like hands get recognized as grabbed).

Combining these two methods also does not solve the issue, which is not surprising given that they both suffer from the same problems.

Now, we do have all the bones of each finger available, with positions and everything. But I have no idea where to start with the math to detect if a finger is curled.

Basically I have this setup for now:

var controller = Leap.loop(function(frame){
   if(frame.hands.length>0){
       //we only look at the first available hand
       var hand = frame.hands[0];
       //we get the index finger only, but later on we should look at all 5 fingers.
       var index = hands.fingers[1];
       //after that we get the positions of the joints between the bones in a hand
       //the position of the metacarpal bone (i.e. the base of your hand)
       var carp = index.carpPosition;
       //the position of the joint on the knuckle of your hand
       var mcp = index.mcpPosition;
       //the position of the following joint, between the proximal and the intermediate bones
       var pip = index.pipPosition;
       //the position of the distal bone (the very tip of your finger)
       var dip = index.dipPosition;

       //and now we need the angle between each of those positions, which is where i'm stuck
   }
});

So, how do I get the angle between two of those positions (carp to mcp, mcp to pip, pip to dip)? Any ideas?

Faience

Answer 1

Alright, I think I found a sort of working approach to detect an actual fist, and not a claw.

First off, instead of the positions of the joints, we need the distance Vectors for each Bone.

Then we calculate the Dot product between the Metacarpal and the Proximal bone, as well as the dot Product between the Proximal and the Intermediate Bone. We can ignore the Distal bone, it doesn't change the result too much.

We sum all the calculated dot products (10 in total) and calculate the average out (we divide by 10). This will give us a value between 0 and 1. A Fist is beneath 0.5 and everything above that is basically not a fist.

Additionally you might also want to check for the amount of extended fingers on a Hand and check if it is 0. This will ensure that a "Thumbs-up" and similiar 1-digit poses do not get recognized as a Fist.

Here is my implementation:

const minValue = 0.5;

var controller = Leap.loop(function(frame){
   if(frame.hands.length>0)
   {
      var hand = frame.hands[0];
      var isFist = checkFist(hand);
   }
});

function getExtendedFingers(hand){
   var f = 0;
   for(var i=0;i<hand.fingers.length;i++){
      if(hand.fingers[i].extended){
         f++;
      }
   }
   return f;
}

function checkFist(hand){
   var sum = 0;
   for(var i=0;i<hand.fingers.length;i++){
      var finger = hand.fingers[i];
      var meta = finger.bones[0].direction();
      var proxi = finger.bones[1].direction();
      var inter = finger.bones[2].direction();
      var dMetaProxi = Leap.vec3.dot(meta,proxi);
      var dProxiInter = Leap.vec3.dot(proxi,inter);
      sum += dMetaProxi;
      sum += dProxiInter
   }
   sum = sum/10;

   if(sum<=minValue && getExtendedFingers(hand)==0){
       return true;
   }else{
       return false;
   }
}

While this works like it should, I doubt that this is the correct and best approach to detect a Fist. So please, if you know of a better way, post it.

---

Solution works perfect, any chance you could explain why you divide by 10 and why the minValue is 0.5? Thanks!

Well, it doesn't work that good, to be honest. I'll soon start to work on a little project that has the goal to improve the detection of fists with Leap Motion.

Regarding your questions, We divide the sum by 10 because we have 2 Bone Joints per finger, with 5 fingers. We want the average value from the sum of all those calculations, because not all fingers will be angled in the same way. So we want some value that encompasses all of these values into a single one: the average value. Given that we have 10 calculations in total (2 per each finger, 5 fingers), we divide the sum of those calculations and there we go. We will get a value between 0 and 1.

Regarding the minValue: Trial&Error. In a project of mine, I used a value of 0.6 instead. This is another problem of this approach: ideally a flat hand should be a value of nearly 0, while a fist should be 1.

Faience

Would this not also pass fingers broken at the joint, pointing the angular-wise opposite direction needed to make a fist (as if completely unrestricted by the range of the joint)? :D. cringe – Fronde

I suppose it would, but this math is a bit too complex for me. :P – Faience

I found a good solution using the distance between each finger and the palm. Calculate the sum of these distances and check whether it's below a certain threshold. – Hernadez

@Hernadez Post what you have made it could be useful for future readers! – Gervais

Answer 2

I know it is an old topic but if you guys still around the answer could be simpler just by using sphereRadius() ;

Erring

Answer 3

I found "grabStrength" is good

Bouillabaisse

He was quite specific about grabStrength not being a good approach for his requirements. This questions is also rather old. If something has changed since 2014 then please be specific about why it is working now. Right now your answer is way too short to be helpful to others... – Ramiroramjet