How to find the contours of a frame from an Android camera and convert them to box2d bodies?

Question

Using OpenFrameworks, OpenCV and Box2D I was able to achieve it with a good framerate. Using Android seems a much more complicated task (partly because I am a JAVA newbie).

This is how I started:

1. Use the "OpenCV Sample - image manipulations" and delete everything except the "canny" effect, which produces a nice black & white image that is perfect to find the contours.

   public Mat onCameraFrame(CvCameraViewFrame inputFrame)
   {
       mRgba = inputFrame.rgba();
       Imgproc.Canny(mRgbaInnerWindow, mIntermediateMat, 50, 100);
       Imgproc.cvtColor(mIntermediateMat, mRgbaInnerWindow, Imgproc.COLOR_GRAY2BGRA, 4);
   
       return mRgba;
   }
   

2. From the "OpenCV Sample - color-blob-detection" I grabbed the logic to find the contours in a Mat:

   // These two lines are actually in the function onCameraViewStarted
   mHierarchy = new Mat();
   CONTOUR_COLOR = new Scalar(255,0,0,255);
   
   // These lines are in function onCameraFrame
   List<MatOfPoint> contours = new ArrayList<MatOfPoint>();        
   Imgproc.findContours(mRgbaInnerWindow, contours, mHierarchy, Imgproc.RETR_EXTERNAL,     
   Imgproc.CHAIN_APPROX_SIMPLE);
   Imgproc.drawContours(mIntermediateMat, contours, -1, CONTOUR_COLOR);
   

   So, my current function looks like this and it doesn't work:

   public Mat onCameraFrame(CvCameraViewFrame inputFrame)
   {
       mRgba = inputFrame.rgba();
   
       if ((mRgbaInnerWindow == null) || (mGrayInnerWindow == null) || (mRgba.cols() != mSizeRgba.width) || (mRgba.height() != mSizeRgba.height))
           CreateAuxiliaryMats();
   
       Imgproc.Canny(mRgbaInnerWindow, mIntermediateMat, 50, 100);
       //Imgproc.cvtColor(mIntermediateMat, mRgbaInnerWindow, Imgproc.COLOR_GRAY2BGRA, 4);
   
       List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
   
       Imgproc.findContours(mRgbaInnerWindow, contours, mHierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
       //Imgproc.drawContours(mIntermediateMat, contours, -1, CONTOUR_COLOR);
   
       return mRgba;
   }
   

3. Now, this is where I'm stucked. I keep getting exceptions and I think I'm not using the right dimensions or transforming the Mat to the right color space. This post has some insight but I don't know if it's correct: OpenCV on Android findContours throws Exception

Foodstuff

What box2d implementation are you using? Is it just bare jbox2d? Or are you using something like ANDEngine? – Guileful

I've been looking at AndEngine but I don't know yet how to implement the CVCamera class with the SimpleGameActivity of AndEngine. – Foodstuff

Answer 1

Hi I am also a newbie in openCV however this code might help,

import android.app.Activity;
import android.os.Bundle;
import android.util.Log;
import android.view.SurfaceView;
import org.opencv.android.*;
import org.opencv.core.*;
import org.opencv.imgproc.Imgproc;


import java.util.ArrayList;
import java.util.List;

public class MainActivity extends Activity implements CvCameraViewListener2{

private static final String TAG = MainActivity.class.getCanonicalName();

private CameraBridgeViewBase mOpenCvCameraView;

private Mat                    mRgba;
private Mat                    mIntermediateMat;
private Mat                    mGray;
Mat hierarchy;


List<MatOfPoint> contours;

private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
            case LoaderCallbackInterface.SUCCESS:
            {
                Log.i(TAG, "OpenCV loaded successfully");
                mOpenCvCameraView.enableView();
            } break;
            default:
            {
                super.onManagerConnected(status);
            } break;
        }
    }
};

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.java_surface_view);
    mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);

    mOpenCvCameraView.setCvCameraViewListener(this);

}

@Override
public void onResume() {
    super.onResume();
    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this,
            mLoaderCallback);
}

@Override
public void onPause() {
    super.onPause();
    if (mOpenCvCameraView != null)
        mOpenCvCameraView.disableView();
}



@Override
public void onCameraViewStarted(int width, int height) {
    mRgba = new Mat(height, width, CvType.CV_8UC4);
    mIntermediateMat = new Mat(height, width, CvType.CV_8UC4);
    mGray = new Mat(height, width, CvType.CV_8UC1);
    hierarchy = new Mat();
}

@Override
public void onCameraViewStopped() {
    mRgba.release();
    mGray.release();
    mIntermediateMat.release();
    hierarchy.release();
}

@Override
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
     mRgba = inputFrame.gray();
        contours = new ArrayList<MatOfPoint>();
        hierarchy = new Mat();

     Imgproc.Canny(mRgba, mIntermediateMat, 80, 100);
     Imgproc.findContours(mIntermediateMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE, new Point(0, 0));
    /* Mat drawing = Mat.zeros( mIntermediateMat.size(), CvType.CV_8UC3 );
     for( int i = 0; i< contours.size(); i++ )
     {
    Scalar color =new Scalar(Math.random()*255, Math.random()*255, Math.random()*255);
     Imgproc.drawContours( drawing, contours, i, color, 2, 8, hierarchy, 0, new Point() );
     }*/
     hierarchy.release();
     Imgproc.drawContours(mRgba, contours, -1, new Scalar(Math.random()*255, Math.random()*255, Math.random()*255));//, 2, 8, hierarchy, 0, new Point());
    // Imgproc.cvtColor(mIntermediateMat, mRgba, Imgproc.COLOR_GRAY2RGBA, 4);
    return mRgba;
}

}

I know this may not be the best way for achieving this, but we are all here to learn new ways :)

Quittor

This causes the Canny detection to keep flashing – Jasso

Yes, someone has a better example to avoid flashing? probably because is in real time. – Efficient

Answer 2

Short answer: You have to use the result of Canny as input (1st argument) for findContours, i.e.

Imgproc.findContours(mIntermediateMat, ...);

Thermosetting