Reusing models from grabcut in OpenCV

Question

I used the interactive grabcut.py from the OpenCV samples to segment an image and saved the foreground and background models. Then I used these models to segment more images of the same kind, as I don't want to retrain the model each time.

After running the grabcut algorithm, the mask is all zeros (all background) and therefore it doesn't segment anything.

from matplotlib import pyplot as plt
import numpy as np
import cv2

img = cv2.imread('usimg1.jpg')
mask = np.zeros(img.shape[:2], np.uint8)
bgdModel = np.load('bgdmodel.npy')
fgdModel = np.load('fgdmodel.npy')

cv2.grabCut(img, mask, None, bgdModel, fgdModel, 5, cv2.GC_EVAL)

mask = np.where((mask==2) | (mask==0), 0, 1).astype('uint8') 
img = img * mask[:, :, np.newaxis]

plt.imshow(img)
plt.show()

I tried to initialize the algorithm with a mask or a rectangle but this produces an error because the models are not empty (which is what I actually want).

How do I have to pass the pre-trained models to the algorithm, such that they are not retrained from scratch each time I'm segmenting an image?

EDIT After rayryeng's comment I implemented following code:

cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 0, cv2.GC_INIT_WITH_RECT)
cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 2, cv2.GC_EVAL)

It seems to work but the first call now changes my model. In the source code it calls learnGMMs without checking whether a pretrained model is provided.

Bauhaus

Bgdmodel.npy is what you want and reuse ? – Compilation

Yes, bgdmodel.npy and fgdmodel.npy – Bauhaus

That's weird... cv2.GC_EVAL's purpose is to already use pretrained models. You can even see that in the source. github.com/opencv/opencv/blob/master/modules/imgproc/src/…. When the GC_EVAL is encountered, it does simple error checking of the masks without actually doing the learning before actually doing the segmentation. You should be able to use cv2.GC_EVAL and specifying the foreground mask to None should work without a problem. – Tredecillion

In addition, the test cases for grabcut also are very curious: github.com/opencv/opencv/blob/…. You see that they first call grabcut with 0 iterations to learn the model, then they reuse those same models again to actually do the segmentation - 2 iterations. Give that a try? Try learning the model first with 0 iterations, then use the precomputed models with GC_EVAL to finally do the segmentation. – Tredecillion

@rayryeng: I tried to call grabCut with 0 iterations and then with 2 (GC_EVAL). It works but during the first call the models are initialized and therefore the pretrained model is lost. I'll add the code to the question. – Bauhaus

@Bauhaus You're right. If you look at the grabcut source even further, you'll see that there is a learnGMMs call that should not be there if you have a pre-trained model: github.com/opencv/opencv/blob/master/modules/imgproc/src/…. The only way to get around this now would be to place an if statement checking for GC_EVAL before the learnGMMs call, then recompile the code. It also looks like this issue has been encountered in the past: #23806564. This really should be fixed. – Tredecillion

BTW initGMMs is only called if you provided the GC_INIT_WITH_RECT or GC_INIT_WITH_MASK flags. It does not get called with GC_EVAL... the culprit is the learnGMMs function which actually does the GMM learning required for Grabcut, but since you have the models already computed, you should not need to do this when the GC_EVAL flag is provided... and so once learnGMMs is called, the models get cleared due to no foreground mask being provided. – Tredecillion

@Tredecillion yes you're right, I fixed it in my question. – Bauhaus

@Bauhaus Understood, so I unfortunately don't have an answer for you other than recompiling the source with the fix I mentioned and generating the Python library again. You have to ask yourself whether it's worth it to recompile the source, which can take several hours for this fix or if you want to go with the burden of recomputing the model for each image that you provide into grab cut. If this is something definitely for long term and it's only applicable to you, then I would probably modify the source. If this is an application generated for public use, then that's another can of worms. – Tredecillion

@ipa: I've opened up an issue. Let's see where this goes: github.com/opencv/opencv/issues/9191 – Tredecillion

@Tredecillion no worries, I want to use in a long term inhouse project, so I guess I'll fix it and recompile the library. – Bauhaus

@Bauhaus Sounds good.... should I write an answer to close things off? – Tredecillion

@Tredecillion yes, would be great. then we can close this one and hope it might be fixed in the future ;) – Bauhaus

@Bauhaus Done! Thanks a lot! – Tredecillion

Just curious. Did doing the fix and recompiling work? – Tredecillion

@Tredecillion yes it worked, thanks! – Bauhaus

@Bauhaus Great to hear! – Tredecillion

Answer 1

You have the correct line of thinking where you use cv2.GC_EVAL so that you only need to perform the segmentation without having to compute the models again. Unfortunately even when you use this flag, this is a limitation with the OpenCV source itself. If you look at the actual C++ implementation when you encounter the GC_EVAL condition, it does this towards the end of the cv::grabcut method. Note that the Python cv2.grabCut method is a wrapper for cv::grabcut:

if( mode == GC_EVAL )
    checkMask( img, mask );

const double gamma = 50;
const double lambda = 9*gamma;
const double beta = calcBeta( img );

Mat leftW, upleftW, upW, uprightW;
calcNWeights( img, leftW, upleftW, upW, uprightW, beta, gamma );

for( int i = 0; i < iterCount; i++ )
{
    GCGraph<double> graph;
    assignGMMsComponents( img, mask, bgdGMM, fgdGMM, compIdxs );
    learnGMMs( img, mask, compIdxs, bgdGMM, fgdGMM );
    constructGCGraph(img, mask, bgdGMM, fgdGMM, lambda, leftW, upleftW, upW, uprightW, graph );
    estimateSegmentation( graph, mask );
}

You'll see that GC_EVAL is only encountered once in the code and that's to check the validity of the inputs. The culprit is the learnGMMs function. Even though you specified the trained models, these get reset because the call to learnGMMs ignores the GC_EVAL flag, so this gets called regardless of whatever flag you specify as the input.

Inspired by this post: OpenCV - GrabCut with custom foreground/background models, what you can do is you'll have to modify the OpenCV source yourself and inside the loop you can place an if statement to check for the GC_EVAL flag prior to calling learnGMMs:

if( mode == GC_EVAL )
    checkMask( img, mask );

const double gamma = 50;
const double lambda = 9*gamma;
const double beta = calcBeta( img );

Mat leftW, upleftW, upW, uprightW;
calcNWeights( img, leftW, upleftW, upW, uprightW, beta, gamma );

for( int i = 0; i < iterCount; i++ )
{
    GCGraph<double> graph;
    assignGMMsComponents( img, mask, bgdGMM, fgdGMM, compIdxs );
    if (mode != GC_EVAL) // New
        learnGMMs( img, mask, compIdxs, bgdGMM, fgdGMM );
    constructGCGraph(img, mask, bgdGMM, fgdGMM, lambda, leftW, upleftW, upW, uprightW, graph );
    estimateSegmentation( graph, mask );
}

This should be able to use the pre-trained models without having to learn them all over again at each iteration. Once you make the change, you'll have to recompile the source again and that should hopefully be able to use your pre-trained models without clearing them when you use the cv2.GC_EVAL flag.

For the future, I have opened up a issue on the official repo for OpenCV. Hopefully they'll fix this when they have the time: https://github.com/opencv/opencv/issues/9191

Tredecillion