Tracking multiple moving objects with KalmanFilter in OpenCV/C++ - How to assign tracks individually to detected objects

Question

I am working on realtime movement detection and object tracking in OpenCV/C++ and am currently stuck at the tracking part.

Matlab-Example of what I want to do: http://www.mathworks.de/de/help/vision/examples/motion-based-multiple-object-tracking.html (I'm troubled with the tracking part and how to transfer it to C++/OpenCV)

My motion part works with OpenCVs BackgroundSubtractor MOG2 which makes it possible to find contours and filter smaller contours out.

For tracking I am currently trying to use the KalmanFilter (with a similar implementation to this) which right now is getting called in every frame if a moving object was found and draws a line on it's path. My Detection & Tracking part looks something like this:

BackgroundSubtractorMOG2 bg;
bg.operator()(frame, threshold);
bg.getBackgroundImage(background);
...  //morphological operations to remove noise etc.
findContours(threshold, ...);
...  //filtering to reject contours which are too smalle/too big

for(int i = 0; i < contours.size(); i++){
approxPolyDP(...);
boundRect = boundingRect(...);
x = boundRect.x + boundRect.width/2;
y = boundRect.y + boundRect.height/2;}

kalmanFilter.track(x,y);
kalmanFilter.draw(frame);

Current Problem: I have a scene with 0 moving objects, then 1 object moves in, gets detected through contours and gets tracked. Then a 2nd object moves in sight, gets detected and makes the tracker jump to it instead of following the first or marking both individually (which I want).

The current tracker takes x & y coordinates of the found object. Like this, once another object gets detected, the tracker still assumes it is the same object but with other coordinates than anticipated.

As one can see, there's no function to assign "tracks" to a certain object which is probably the biggest problem. I read about the hungarian algorithm but am not quite sure on how to implement it within my functions.

What would be a good method to make the tracking work for multiple objects?

My idea was that if I had each object uniquely identified I could check if the ID is still the same and if not let the tracker know that it's a new object to track separately from the other. Not sure if this is necessary or even useful though and if so, how to do it.

Agnesse

can you draw an image and mark some comments inside to explain your problem? assigning individual IDs is no problem, but I guess your problem is the registration of contours/object between frames?!? – Malady

edited the question and explained the problem a little further. hope it makes it more understandable. – Agnesse

hard to help if you dont post your code... I guess you're using "tracking by detection"? so you "detect" with backgroundsubtraction and you use the tracking only for "smoothing" the movement? You'll want to use some real tracking where the tracking itself identifies the location of the target, independently from the detection of new targets. If you really want to use simple tracking by detection, you'll first want to cluster each detection in each frame (e.g. the detected object is the same as the shortest distance object in last frame and you create one tracker for each object) – Malady

edited the question again as I searched a little more. Also put in some code to explain my motion detected part and the kalman-tracking algorithm. I want to detect moving objects (which I already do) and then track them further on, e.g. marking them with differently colored bounding boxes & maybe a number in it. See the example link I posted where the exact same thing is done in Matlab. And yes, your last sentence is probably how it's gonna be done (using a certain way to calculate best matches through distance). I'd love to have a c++/opencv example to see how I can fit it in my code. – Agnesse

what is boundingRect? the bounding box of a single contour or bb of all contours?!? do you use the kalman filter only for the last contour?!? – Malady

Ah yes sorry. The boundingRect is called in a loop iterating through the vector containing the contours. (adding that to the code) Right now, the Kalmanfilter is only used for the last contour of each frame as it gets even more messed up if called inside the loop as it just jumps between every contour then. If the assigning part is done it will probably have to be called within the loop but I'm still one step behind that. – Agnesse

@Agnesse Hello did you do it?? How is that you assigned kalman filter to to each centroid.!! Even i'm struck here. – Koffman

hey. it's like Micka said in his answer. a new kalman filter gets created for every new found centroid. Meaning that I create and delete objects on certain conditions and create new KFs whenever I create a new object – Agnesse

Answer 1

try something like this:

for each contour:
    if its already tracked with ID 'A': kalmanFilter_with_id_A.track(x,y);
    else createNewKalmanFilterWithID A

you need some mechanism to decide whether it is already tracked. In simple tracking you just decide by measuring distances to contours in last frame and if its close enough, it's the old object. This is quite erroneous so you might want to have a look at better tracking methods, e.g. probabilistic tracking.

so simple mode:

for each contour 'currentFrameC':
    for each contour 'lastFrameC'
        if distance(currentFrameC, lastFrameC) is smallest and < threshold
            currentFrameC is the same object as lastFrameC so give it the same ID
    if no shortest contour with dist < thres was found, create a new object with a new ID and create a new KalmanFilter with this same ID for that object

call kalmanFilter with ID for each found contour ID

Malady

Yeah, my thinking was pretty much the same so far I suppose, I just wasn't sure on how to go with the ID-part. Guess I'll have to write another class specifying the tracking targets to get IDs assigned to it. And yes, the tracking might have to get refined afterwards but for now it's all I need. I'll accept your answer as it's probably still the closest to what I want. Thanks for the help. – Agnesse