Calculate BPM from Kinect sensor data

Question

I am struggeling with the Kinect for Windows SDK to create an application for conducting (with C#).

Basically I need to track one hand (usually the right one) of a conductor and recognize his speed in directing (BPM) to send this value to another application via MIDI.

What I started with is the SkeletonFramesReadyEvent adding the JointType.HandRight with a DateTime.Now.Ticks timestamp to a history List which is updated and removes the first entry. I keep the history of 60 frames (2 seconds).

I calculate the BPM by searching for the last low and high of the Joint.Position.Y and then calculate the difference and divide bpm = 60*ticksPerSecond/diff. However the result is wrong. Is there another way of doing this? What am I missing?

This is what I am using so far:

public int DetectBPM(JointType type)
{
    // we have not history yet
    if (!HasHistory()) return 0;

    // only calculate every second
    var detectTime = DateTime.Now.Second;
    if (_lastBPM != 0 && _lastBPMDectect == detectTime) return _lastBPM;

    // search last high/low boundaries
    var index = (int) type;
    var list = History[index];
    var i = list.Count - 1;

    var lastHigh = list[i];
    var lastLow = list[i];

    // shift to last peak first
    while (i > 0 && list[i].Joint.Position.Y >= list[i - 1].Joint.Position.Y) i--;

    // find last low
    while (i >= 0 && lastLow.Joint.Position.Y >= list[i].Joint.Position.Y) lastLow = list[i--];

    // find last high
    while (i >= 0 && lastHigh.Joint.Position.Y <= list[i].Joint.Position.Y) lastHigh = list[i--];

    var ticks = lastLow.Timestamp - lastHigh.Timestamp;
    var elapsedTime = new TimeSpan(ticks);

    var bpm = (int) (60000/elapsedTime.TotalMilliseconds);

    Console.WriteLine("DEBUG: BPM = " + _lastBPM + ", elapsedMS: " + elapsedTime.TotalMilliseconds);

    _lastBPMDectect = detectTime;
    _lastBPM = bpm;

    return _lastBPM;
}

Alton

What result are you getting and what result are you expecting? – Adallard

DEBUG: BPM = 512, elapsedMS: 328 DEBUG: BPM = 182, elapsedMS: -322 DEBUG: BPM = -186, elapsedMS: -337 DEBUG: BPM = -178, elapsedMS: 299 DEBUG: BPM = 200, elapsedMS: 683 DEBUG: BPM = 87, elapsedMS: -378 DEBUG: BPM = -158, elapsedMS: 92 – Alton

First, you should probably be using Math.Abs() around the difference in the calculation of ticks (that'll account for the negative values). According to the documentation Timestamp is in milliseconds, not ticks, so use TimeSpan.FromMilliseconds(ticks) instead of new TimeSpan(ticks) (probably rename ticks too) – Adallard

If I subtract an older timestamp from a younger, a negative value should not occur. Second, Timestamp is in ticks - this is my own class which holds a Joint with a Timestamp in ticks. So this does not solve my problem. – Alton

My apologies on ticks then, your question is tagged kinect, hence the assumption that the kinect sdk docs applied. – Adallard

Can you add an example graph of your history data over time? – Sylvanite

Well therefor I'd have to figure out how to plot data with c# first. – Alton

Just output your data to a file in CSV format and plot it in Excel... – Clinician

Your finding the Beats per Minute of a person using a gesture? – Respect

Exactly. Basically a person does a typical up-and-down gesture with one hand. I want to - in the simplest version - track the time between to two lowest points of his/her hand which represent a beat. With the time difference between two beats I should be able to calculate the BPM. – Alton

Answer 1

I figured out how to do it. I was missing a point and calculated the BPM between a peak and a low position of the hand which is wrong. I have to calucalte the time difference between the last two low points to get the correct result.

The correct way to go is, find the stating point which is the last peak. From there move to the last low, this is the first point to calculate the difference from. The move to the next peak and go down to the next low again which is the second point to calculate the difference from.

The principle is shown in the figure below

And that results in a nicely BPM calculated like this:

var ticks = Math.Abs(firstLow.Ticks - secondLow.Ticks);
var elapsedTime = new TimeSpan(ticks);

var bpm = (int) (60000/elapsedTime.TotalMilliseconds);

Thanks for participating anyway.

Alton

If your data is noisy that detection of a minima is going to be suspect. You may well get local minima just one tick apart. This is why I am saying you probably need to smooth your data, (Its possible Kinect is smoothing it for you - but without seeing a graph of the data its hard to tell.) Its also quite likely that with mild noise that what you've got will work most of the time - then fail. – Sylvanite