I am processing digital signals, trying to detect fingertips, similar to the solution that is presented here: Hand and finger detection using JavaCV.
However, I am not using JavaCV but OpenCV for android which is slightly different. I have managed to do all the steps presented in the tutorial, but filtering of convex hulls and convexity defects. This is how my image looks like:
Here is an image in another resolution:
As you can clearly see, There is to many yellow points (convex hulls) and also to many red points (convexity deffects). Sometimes between 2 yellow points there is no red point, which is quite strange (how are convex hulls calculated?)
What I need is to create simillar filtering function like in the link provided before, but using data structures of OpenCV.
Convex Hulls are type of MatOfInt ... Convexity defects are type of MatOfInt4 ...
I created also some additional data structures, because stupid OpenCV uses different types of data containing same data, in different methods...
convexHullMatOfInt = new MatOfInt();
convexHullPointArrayList = new ArrayList<Point>();
convexHullMatOfPoint = new MatOfPoint();
convexHullMatOfPointArrayList = new ArrayList<MatOfPoint>();
Here is what I did so far but it is not working good. The problem is probably with converting data in a wrong way:
Creating convex hulls and convexity defects:
public void calculateConvexHulls()
{
convexHullMatOfInt = new MatOfInt();
convexHullPointArrayList = new ArrayList<Point>();
convexHullMatOfPoint = new MatOfPoint();
convexHullMatOfPointArrayList = new ArrayList<MatOfPoint>();
try {
//Calculate convex hulls
if(aproximatedContours.size() > 0)
{
Imgproc.convexHull( aproximatedContours.get(0), convexHullMatOfInt, false);
for(int j=0; j < convexHullMatOfInt.toList().size(); j++)
convexHullPointArrayList.add(aproximatedContours.get(0).toList().get(convexHullMatOfInt.toList().get(j)));
convexHullMatOfPoint.fromList(convexHullPointArrayList);
convexHullMatOfPointArrayList.add(convexHullMatOfPoint);
}
} catch (Exception e) {
// TODO Auto-generated catch block
Log.e("Calculate convex hulls failed.", "Details below");
e.printStackTrace();
}
}
public void calculateConvexityDefects()
{
mConvexityDefectsMatOfInt4 = new MatOfInt4();
try {
Imgproc.convexityDefects(aproximatedContours.get(0), convexHullMatOfInt, mConvexityDefectsMatOfInt4);
if(!mConvexityDefectsMatOfInt4.empty())
{
mConvexityDefectsIntArrayList = new int[mConvexityDefectsMatOfInt4.toArray().length];
mConvexityDefectsIntArrayList = mConvexityDefectsMatOfInt4.toArray();
}
} catch (Exception e) {
Log.e("Calculate convex hulls failed.", "Details below");
e.printStackTrace();
}
}
Filtering:
public void filterCalculatedPoints()
{
ArrayList<Point> tipPts = new ArrayList<Point>();
ArrayList<Point> foldPts = new ArrayList<Point>();
ArrayList<Integer> depths = new ArrayList<Integer>();
fingerTips = new ArrayList<Point>();
for (int i = 0; i < mConvexityDefectsIntArrayList.length/4; i++)
{
tipPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i]));
tipPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i+1]));
foldPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i+2]));
depths.add(mConvexityDefectsIntArrayList[4*i+3]);
}
int numPoints = foldPts.size();
for (int i=0; i < numPoints; i++) {
if ((depths.get(i).intValue()) < MIN_FINGER_DEPTH)
continue;
// look at fold points on either side of a tip
int pdx = (i == 0) ? (numPoints-1) : (i - 1);
int sdx = (i == numPoints-1) ? 0 : (i + 1);
int angle = angleBetween(tipPts.get(i), foldPts.get(pdx), foldPts.get(sdx));
if (angle >= MAX_FINGER_ANGLE) // angle between finger and folds too wide
continue;
// this point is probably a fingertip, so add to list
fingerTips.add(tipPts.get(i));
}
}
Results (white points - fingertips after filtering):
Could you help me to write proper function for filtering?
UPDATE 14.08.2013
I use standard openCV function for contour approximation. I have to change approximation value with resolution change, and hand-to-camera distance, which is quite hard to do. If the resolution is smaller, then finger consist of less pixel, thus approximation value should be lover. Same with the distance. Keeping it high will result in completely losing the finger. So I think approximation is not good approach to resolving the problem, however small value could be useful to speed up calculations:
Imgproc.approxPolyDP(frame, frame, 2 , true);
If I use high values, then the result is like on the image below, which would be good only if distance and resolution wouldn't change. Also, I am quite surprised that default methods for hulls points and defects points doesn't have useful arguments to pass (min angle, distance etc)...
Image below presents the effect that I would like to achieve always, independently from resolution or hand-to-camera distance. Also I don't want to see any yellow points when I close my palm...
To sum everything up, I would like to know:
- how to filter the points
- how can I make resolution and distance independent approximation which will always work
- if someone knows or have some materials (graphical representation, explanation) about those data structures used in OpenCV, I would be happy to read it. (Mat, MatOfInt, MatOfPoint, MatOfPoint2, MatOfPoint4 etc.)