Kinect pattern recognition [closed]

Question

I bought a Kinect to play around with on my PC in C#. (And what fun it is to program!)

I'm using CLNUIDevice.dll to get the depth image from Kinect. This works fine and I'm able to read the depth through the RGB values in the image.

Now I'm wondering what libraries there are to do different types of recognition. I've seen hand recognition, face recognition, skeleton, emotions, objects, etc.

The image processing libraries don't have to be for Kinect (though that would be nice) and they don't have to be for .Net (though that would be nice). For instance: Any object tracking lib will work, but if it can use the 3D heatmap from Kinect properly it will work much better.

Hieroglyphic

All the demos I've seen use OpenCV. It's a C library with an ugly API but there should be C# bindings. – Bubbler

Answer 1

You can take the series of RGB matrices produced by the Kinect and run them through standard image processing algorithms, in practice image processing algorithms are normally combined together to produce meaningful results. Here are a few standard techniques that could easily be implemented ( and combined) in .net:

Template Matching - a technique in digital image processing for finding small parts of an image which match a template image http://en.wikipedia.org/wiki/Template_matching

Morphological Image Processing - a theory and technique for the analysis and processing of geometrical structures, based on set theory, lattice theory, topology, and random functions http://ashleyaberneithy.wordpress.com/2011/08/08/automating-radiology-detecting-lung-nodules-using-morphological-image-processing-in-f/

There are also more advanced image processing techniques that can be used in specific scenarios, e.g face recognition and pattern matching via machine learning

Principle Component Analysis - I've used this technique in the past, and I think that this is used in modern consumer cameras to perform facial recognition http://en.wikipedia.org/wiki/Principal_component_analysis

Machine Learning pattern matching - I've used Support Vector Machines and Neural Network based learning algorithms in the past to detect patterns in image matrices. It's worth reading Vapnik's Statistical Learning Theory - http://www.amazon.com/Statistical-Learning-Theory-Vladimir-Vapnik/dp/0471030031 which shows how to successfully map training data into an n-dimensional structure, and how to successfully model hyperplanes within the structure which classify data, new data can then be classified based on this model. A library called LibSVM also exists which I have found useful. http://www.csie.ntu.edu.tw/~cjlin/libsvm/

Just a side note, it would probably be more natural to use F# within the .net world to implement some of these algorithms

EDIT : another really good book is "Digital Image Processing"

Ivonneivor

Answer 2

A useful .Net library for computer vision and machine learning is AForge.NET

Here is a code sample and a video of someone using it with Kinect: Conquex

Benison

Answer 3

Learn OpenCV. It is a standard framework for computer vision labs/learning today. Try contour finding facilities, K-nearest neighbors, etc. But first, learn its interface (data structures, functions).

Bushwhacker

Answer 4

And the oblictory OpenCV and my fav port to C# http://www.emgu.com/wiki/index.php/Main_Page

Barbarese

Answer 5

Siding with Kenny on this one - the Emgu C# wrapper to OpenCV is what I'd go for. OpenCV (usually based in C++) is the most widely used Computer Vision framework out there, and is also partially maintained/optimized by Intel. A lot of current researchers and engineers use it to create their own projects, and the Emgu wrapper give's you a great translation to start making use of its libraries in C#. I feel that a lot of the open source innovation in the Kinect area is happening with the libfreenect wrappers (best used with Linux/C++) and EmguCV should give you the best way to imitate (or use others' imitations) of what these Vision researchers are doing.

Dement

Answer 6

Some correction about OpenCV, it used to be a C library, while the legacy code is still being maintained, OpenCV is now almost purely C++ i.e. from version 2.1 onwards. A lot of which was really complicated to deal with when it was a C only library has been simplified in the newer versions.

Also there is a Yahoo OpenCV group, which works as a great resource. Also openCV comes with a set of samples which help you a great deal with respect to getting grips with basic pattern matching.

A link to the Yahoo groups http://tech.groups.yahoo.com/group/OpenCV/

A sample of Kinect and OpenCV http://www.morethantechnical.com/2010/11/22/kinect-and-opencv-2-1/

Elkeelkhound