opencv4android template matching using camera

Question

I have downloaded and successfully run the example provided in opencv4android sdk.

I am able to simply display the camera frames without any processing,

 public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
     return inputFrame.rgba();
 }

I want to process live frame with some predefined image template to recognize that template. I have taken reference from this post and implemented accordingly. But I get black screen only.

private Mat mCameraMat = new Mat();
private Mat mTemplateMat;

 public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    mCameraMat = inputFrame.rgba();
    initialize();

     int match_method = Imgproc.TM_SQDIFF;

        // Create the result matrix
        int result_cols = mCameraMat.cols() - mTemplateMat.cols() + 1;
        int result_rows = mCameraMat.rows() - mTemplateMat.rows() + 1;
        Log.d(TAG, " mCameraMat cols "+mCameraMat.cols());
        Log.d(TAG, " mCameraMat rows "+mCameraMat.rows());
        Log.d(TAG, " mTemplateMat cols "+mTemplateMat.cols());
        Log.d(TAG, " mTemplateMat rows "+mTemplateMat.rows());

       Mat result = new Mat(result_rows, result_cols, CvType.CV_32F);

        // Do the Matching and Normalize
        Imgproc.matchTemplate(mCameraMat, mTemplateMat, result, match_method);



        Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());

        // Localizing the best match with minMaxLoc
        MinMaxLocResult mmr = Core.minMaxLoc(result);

        Point matchLoc;
        if (match_method == Imgproc.TM_SQDIFF || match_method == Imgproc.TM_SQDIFF_NORMED) {
            matchLoc = mmr.minLoc;
        } else {
            matchLoc = mmr.maxLoc;
        }

        Rect roi = new Rect((int) matchLoc.x, (int) matchLoc.y, mTemplateMat.cols(), mTemplateMat.rows());
        Core.rectangle(mCameraMat, new Point(roi.x, roi.y), new Point(roi.width - 2, roi.height - 2), new Scalar(255, 0, 0, 255), 2);           
 return result;
}

public void initialize(){

    try {
        if (mCameraMat.empty())
            return;
        if(mTemplateMat == null){
            Mat temp = Utils.loadResource(Tutorial1Activity.this, R.drawable.icon);
            mTemplateMat = new Mat(temp.size(), CvType.CV_32F);
            Imgproc.cvtColor(temp, mTemplateMat, Imgproc.COLOR_BGR2RGBA);
            Log.d(TAG, "initialize mTemplateMat cols "+mTemplateMat.cols());
            Log.d(TAG, "initialize mTemplateMat rows "+mTemplateMat.rows());
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}

Note:

My ultimate goal is to recognize the playing cards from live camera. Kindly suggest best approach. Should I use image templates or any other thing to make things faster?

This is how I want to recognize multiple cards from live camera:

Result should be: ♠A ♠K ♠Q ♠J ♠10 when camera preview seems like below

Arnulfoarny

It seems to me your onCameraFrame has two return statements, am I wrong? – Sagerman

@AlessandroJacopson: I did mistake while posting code in the question. I have corrected it now. – Arnulfoarny

I am not able to develop on Android. It seems to me that the maximu gray level in image result will be 1 after the call to Core.normalize, am I right? If yes, then on the range 0=black and 255=white you will have that result is almost black. Furthermore result is of type CV_32F while maybe you need a 3 channel image (R, G, B as you wrote in your first code snippet which you say it is working fine). And your Core.rectangle will draw on the image mCameraMat but you do not return mCameraMat... – Sagerman

what are the respective sizes of the images? I would start by converting both images to 8bit and checking the result. You can save the resulting image and look at that, as Alessandro J pointed out, its is likely black. – Billion

size of template image is 72x72. Even when I skip normalize function, I get black screen.One more doubt I have in my mind, I want recognize playing cards, so is it possible to checkout multiple cards at same time? or do I have to implement something else? – Arnulfoarny

@MehulJoisar What do you get if you use Core.normalize(result, result, 0, 255, Core.NORM_MINMAX, -1, new Mat()); ? – Sagerman

@AlessandroJacopson: Still I am facing the same. – Arnulfoarny

so the template image is 72x72? and what is the size of the image you are checking? Are they both the same format? Make things even easier for yourself to test your code. Take an image, and crop a section out of it. Make that your template and run it through your code, that should work if your code is ok. There is an art to selecting the template images, but first ensure your code works. – Billion

@don_q: yes I had cropped that template from original image itself.original image size is 1080x1920. – Arnulfoarny

Answer 1

Template matching is unlikely to be the best approach here.

1. Try aSIFT to do an affine invariant SIFT matching or a normal SIFT (OpenCV implementation exists). However, since these are in C++, you may want to use JNI to make calls to it from Java on an Android device. This is probably the best way to detect the suit of the card from the 4 symbols.
2. Another option to detect and recognize the numbers/alphabets on the cards is to use a text detector like MSER and then a text recognizer on the regions of interest indicated by the MSER filter.

In any case, you are unlikely to be able produce the best of results from the kind of image you've shown in the image. You may be able to get acceptable performance for full frontal, upright images with the first method.

Surat