Get background model from BackgroundSubtractorMOG2 in python

Question

I need to get the background model of a Mixture of Gaussian with opencv. I know that there is a method called getBackgroundImage in C++ I searched if it is possible to get it in python interface but I haven't get good result. I Tried opencv 3.0.0-dev because it has BackgroundSubtractorMOG2 implementation, but help() function don't document method implementation for background model. Do you know if there is undocumented implementation? I searched how to edit opencv source to implement a python implementation but i haven't found documentation about it. I prefer avoid to use scipy.weave to compile c++ code, furthermore i don't know if scipy.weave is useful in thi situation

Tristis

try this: opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/… – Rufford

I'm using opencv 3.0.0-dev because I already readed link that you suggest...Now i'm trying to understand how opencv python parser works...otherwise if there aren't other solution, I think that I should write a wrapper of MOG2 with Python C API – Tristis

perhaps something like this is sufficient? bg = 0.99*bg+0.01*(frame& ~cv2.cvtColor(fg,cv2.COLOR_GRAY2BGR)); cv2.imshow('bg',bg/255) – Oxtail

MOG2 bindings are already available in opencv 3-dev. Then why you write your own? – Rufford

@AbidRahmanK MOG2 binding in opencv 3-dev haven't getBackgroundImage that I need to work. – Tristis

@ZawLin It isn't a perfect approximation of background model, it return me a black color in many points. Thanks for this suggest. – Tristis

@EmanuelOverflow: Sorry to ask, but what do you mean by backgroundimage? Isn't it opposite of foreground image that you get? – Rufford

@AbidRahmanK No the background image is the model used by MOG to update the foreground mask. – Tristis

Answer 1

Adapted Zaw Lin's solution on

• Ubuntu 18.04
• OpenCV 3.2 installed via apt install libopencv-dev

The main difference is that the result (fg / bg) images are created/allocated in python and then passed down to the c++ lib. Zaw Lin's solution was giving me errors (errno 139 - SIG_SEGV), because of the app was accessing invalid memory zones. Hope it saves someone a couple of hours :)

mog2.cpp:

#include <opencv2/opencv.hpp>

cv::BackgroundSubtractorMOG2 *mog = cv::createBackgroundSubtractorMOG2 (500, 16, false);

extern "C" void getfg(int rows, int cols, unsigned char* imgData,
        unsigned char *fgD) {
    cv::Mat img(rows, cols, CV_8UC3, (void *) imgData);
    cv::Mat fg(rows, cols, CV_8UC1, fgD);
    mog->apply(img, fg);
}

extern "C" void getbg(int rows, int cols, unsigned char *bgD) {
    cv::Mat bg = cv::Mat(rows, cols, CV_8UC3, bgD);
    mog->getBackgroundImage(bg);
}

Compile it like:

gcc  \
    -shared \
    -o libmog2.so  \
    -fPIC ./mog2.cpp  \
    -lopencv_core -lopencv_highgui -lopencv_objdetect -lopencv_imgproc -lopencv_features2d -lopencv_ml -lopencv_calib3d -lopencv_video

And then python:

mog2.py

import numpy as np
import ctypes as C
import cv2

libmog = C.cdll.LoadLibrary('path/to/libmog2.so')

def getfg(img):
    (rows, cols) = (img.shape[0], img.shape[1])
    res = np.zeros(dtype=np.uint8, shape=(rows, cols))
    libmog.getfg(img.shape[0], img.shape[1],
                       img.ctypes.data_as(C.POINTER(C.c_ubyte)),
                       res.ctypes.data_as(C.POINTER(C.c_ubyte)))
    return res


def getbg(img):
    (rows, cols) = (img.shape[0], img.shape[1])
    res = np.zeros(dtype=np.uint8, shape=(rows, cols, 3))

    libmog.getbg(rows, cols, res.ctypes.data_as(C.POINTER(C.c_ubyte)))
    return res


if __name__ == '__main__':
    c = cv2.VideoCapture(0)
    while 1:
        _, f = c.read()
        cv2.imshow('f', f)
        cv2.imshow('fg', getfg(f))
        cv2.imshow('bg', getbg(f))
        if cv2.waitKey(1) == 27:
            exit(0)

Noyes

thanks, i tested it with a statically compiled version of opencv. so was taking a while to test it with shared libraries. – Oxtail

re Emanuel: corrected Bug: shape=(rows, cols, 1) in getfg should be only shape=(rows,cols). This caused bugs if I wrapped (transformed) res into a SimpleCV Image ( with Image(res)). Updated answer – Noyes

@Matyas, I am not able compile c++ code to libmog2.so file. I am keep getting error 'opencv2/opencv.hpp: No such file or directory'. I am using windows 10. I tried adding #include "C:/opencv2/opencv.hpp" but no luck. Could you please share how you solved this issue? – Terrigenous

Hey @Vendetta, I've rechecked the compilation on Ubuntu18.04;OpenCV3.2 updated the post so C++ compilation works. You have a more generic issue, where your installed header files are not found/detected so that is what you should look into on how to configure that in windows. I haven't gone through these steps since on linux/ubuntu works out of the box. – Noyes

Answer 2

here's a simple wrapper using ctypes, i have only tested on windows

cpp, build as dll

#include "opencv2/opencv.hpp"
cv::BackgroundSubtractorMOG2 mog(100, 16, false);

cv::Mat bg;
cv::Mat fg;
extern "C" __declspec(dllexport)  unsigned char*  getfg(int rows,int cols, unsigned char* fdata)
{
    cv::Mat frame= cv::Mat(rows, cols, CV_8UC3,fdata);
    mog(frame,fg);
    //check fg.iscont(), copy as needed
    return fg.data;
}


extern "C" __declspec(dllexport)   unsigned char*  getbg()
{
    mog.getBackgroundImage(bg);
    return bg.data;
}

python

import cv2
import numpy as np
import ctypes as C
lib = C.cdll.LoadLibrary('wrapper.dll')

def getfg(img):
    ptr = lib.getfg(img.shape[0],img.shape[1],img.ctypes.data_as(C.POINTER(C.c_ubyte)))

    buf = (C.c_ubyte * img.shape[0] * img.shape[1]  * 1).from_address(ptr)
    res = np.ndarray(buffer=buf, dtype=np.uint8,
                       shape=(img.shape[0], img.shape[1], 1))
    return res

def getbg(img):
    ptr = lib.getbg()
    buf = (C.c_ubyte * img.shape[0] * img.shape[1]  * 3).from_address(ptr)
    res = np.ndarray(buffer=buf, dtype=np.uint8,
                       shape=(img.shape[0], img.shape[1], 3))
    return res

c = cv2.VideoCapture(0)
while(1):
    _,f = c.read()
    cv2.imshow('f',f)
    cv2.imshow('fg',getfg(f))
    cv2.imshow('bg',getbg(f))
    if cv2.waitKey(1)==27:
        exit(0)    

Oxtail

I'm on Mac OS X, I deleted __declspec(ddlexport) and created a dylib. ctypes works but I get segmentation fault on imshow of getfg(f) – Tristis

it's gonna take me a while to get my hands on a mac and i am not familiar with mac programming but i tried it on ubuntu and it works fine. can you try it on ubuntu and if you still have problem then i can update with more details – Oxtail

I have written a wrapper with Python C API, but your is the best solution. I'll try to force it to work on my system :) – Tristis

Removing __declspec(dllexport) makes it compilable on Ubuntu (as shared lib with all opencv libs added with -l). Though my problem is that I cannot load the image from the returned address. The line np.ndarray(buffer=buf[...] fails with SIG_SEGV (segmentation fault). Any ideas? – Noyes

@ZawLin, I am literally trying on windows for long time now to build wrapper.dll file. Its just not working. I installed opencv and tried to compile using this command gcc -o fgbg.dll mog2.cpp -I C:/opencv3.4.10/opencv/build/include -I C:/opencv3.4.10/opencv/build/include/opencv2 -L C:/opencv3.4.10/opencv/build/x64/vc14/lib it gives me errors like these ` undefined reference to cv::String::allocate(unsigned int)' c:/mingw/bin/../lib/gcc/mingw32/9.2.0/../../../../mingw32/bin/ld.exe: Could you please explain how build wrapper.dll? – Terrigenous

Hi that was ages ago, I don't quite remember. But I was using visual studio so things were a little different. But guessing from your error, it sounds like you are missing some dependencies to link that you need to specify in the compiler. Like maybe -llibopencv_core.lib etc. You should make sure you can compile an opencv exeuctable normally first. – Oxtail

@ZawLin, Thank you very much for prompt response. I tested this code by compiling through visual studio, it compiles fine. But I am not sure how can I covert this cpp file to dll using visual studio. That is why I was troubleshooting with gcc. – Terrigenous

Answer 3

opencv 3.0

bgd=dict(history=20,nmixtures=20,backgroundRatio=0.5,noiseSigma=0)
fgbg=cv2.bgsegm.createBackgroundSubtractorMOG(**bgd)

Cottar