opencv background substraction

Question

I have an image of the background scene and an image of the same scene with objects in front. Now I want to create a mask of the object in the foreground with background substraction. Both images are RGB.

I have already created the following code:

cv::Mat diff;
diff.create(orgImage.dims, orgImage.size, CV_8UC3);
diff = abs(orgImage-refImage);

cv::Mat mask(diff.rows, diff.cols, CV_8U, cv::Scalar(0,0,0));
//mask = (diff > 10);

for (int j=0; j<diff.rows; j++) {
    // get the address of row j
    //uchar* dataIn= diff.ptr<uchar>(j);
    //uchar* dataOut= mask.ptr<uchar>(j);
    for (int i=0; i<diff.cols; i++) {
        if(diff.at<cv::Vec3b>(j,i)[0] > 30 || diff.at<cv::Vec3b>(j,i)[1] > 30 || diff.at<cv::Vec3b>(j,i)[2] > 30)
            mask.at<uchar>(j,i) = 255;
    }
}

I dont know if I am doing this right?

Krystalkrystalle

Answer 1

Have a look at the inRange function from OpenCV. This will allow you to set multiple thresholds at the same time for a 3 channel image.

So, to create the mask you were looking for, do the following:

inRange(diff, Scalar(30, 30, 30), Scalar(255, 255, 255), mask);

This should also be faster than trying to access each pixel yourself.

EDIT : If skin detection is what you are trying to do, I would first do skin detection, and then afterwards do background subtraction to remove the background. Otherwise, your skin detector will have to take into account the intensity shift caused by the subtraction.

Check out my other answer, about good techniques for skin detection.

EDIT :

Is this any faster?

int main(int argc, char* argv[])
{
    Mat fg = imread("fg.jpg");
    Mat bg = imread("bg.jpg");

    cvtColor(fg, fg, CV_RGB2YCrCb);
    cvtColor(bg, bg, CV_RGB2YCrCb);

    Mat distance = Mat::zeros(fg.size(), CV_32F);

    vector<Mat> fgChannels;
    split(fg, fgChannels);

    vector<Mat> bgChannels;
    split(bg, bgChannels);

    for(size_t i = 0; i < fgChannels.size(); i++)
    {
        Mat temp = abs(fgChannels[i] - bgChannels[i]);
        temp.convertTo(temp, CV_32F);

        distance = distance + temp;
    }


    Mat mask;
    threshold(distance, mask, 35, 255, THRESH_BINARY);

    Mat kernel5x5 = getStructuringElement(MORPH_RECT, Size(5, 5));
    morphologyEx(mask, mask, MORPH_OPEN, kernel5x5);

    imshow("fg", fg);
    imshow("bg", bg);
    imshow("mask", mask);

    waitKey();

    return 0;
}

This code produces this mask based on your input imagery:

Finally, here is what I get using my simple thresholding method:

    Mat diff = fgYcc - bgYcc;
    vector<Mat> diffChannels;
    split(diff, diffChannels);

    // only operating on luminance for background subtraction...
    threshold(diffChannels[0], bgfgMask, 1, 255.0, THRESH_BINARY_INV);

    Mat kernel5x5 = getStructuringElement(MORPH_RECT, Size(5, 5));
    morphologyEx(bgfgMask, bgfgMask, MORPH_OPEN, kernel5x5);

This produce the following mask:

Culbertson

Thanks for the comment! but am I doing this right? Because there are pixels in the diff image that are totaly black at places where there is a difference between the input image and the reference image... – Krystalkrystalle

Can you post the images that you are using in the above algorithm? – Culbertson

This is the image:dl.dropbox.com/u/5276376/image.jpg The reference image: dl.dropbox.com/u/5276376/refImage.jpg and the difference: dl.dropbox.com/u/5276376/diff.jpg (yes there is a small difference in illumination between the image and the reference image). As you can see the arm in the difference image contains many black pixels. As a resulall these pixels are lost with a threshold of e.g. 30 – Krystalkrystalle

Hmm I think the problem is that I'm using the RGB colorspace which is not a perceptually uniform color space. Two colors might look very similar while two other colors separated by the same distance will look very different... So now I fist converted the images to the YCrCb colorspace but the results are even worse. Maybe it is because I'm using the wrong type to save each pixel and therefore values below zero will be clipped to zero or something?? Currently I'm using a CV_32F for the difference: diff.create(table.dims, table.size, CV_32F); diff = abs(table - refRoi); – Krystalkrystalle

I know the skin detection technique but I also want the contour of other parts such e.g. clothes... so therefore the background substraction technique – Krystalkrystalle

Answer 2

I think when I'm doing it like this I get the right results: (in the YCrCb colorspace) but accessing each px is slow so I need to find another algorithm

    cv::Mat mask(image.rows, image.cols, CV_8U, cv::Scalar(0,0,0));

    cv::Mat_<cv::Vec3b>::const_iterator itImage= image.begin<cv::Vec3b>();
    cv::Mat_<cv::Vec3b>::const_iterator itend= image.end<cv::Vec3b>();
    cv::Mat_<cv::Vec3b>::iterator itRef= refRoi.begin<cv::Vec3b>();
    cv::Mat_<uchar>::iterator itMask= mask.begin<uchar>();

    for ( ; itImage!= itend; ++itImage, ++itRef, ++itMask) {
        int distance = abs((*itImage)[0]-(*itRef)[0])+
                        abs((*itImage)[1]-(*itRef)[1])+
                        abs((*itImage)[2]-(*itRef)[2]);

        if(distance < 30)
            *itMask = 0;
        else
            *itMask = 255;
    }

Krystalkrystalle