Android draw on camera preview

Question

I'm making a virtual reality application where the camera should detect faces, locate them and show their location on the camera preview.

I know of 3 ways to do it, I'd like to use GLSurfaceView to be as fast as possible (according to this post), but currently I'm trying to draw on the same SurfaceView where the camera is using for its preview. My callback to draw on it would be onFaceDetection like so:

public class MyActivity extends Activity implements SurfaceHolder.Callback, Camera.FaceDetectionListener {
    Camera camera;
    SurfaceView svPreview;
    SurfaceHolder previewHolder;
    TextView tvInfo;
    Paint red;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        svPreview = (SurfaceView) findViewById(R.id.svPreview);
        tvInfo = (TextView) findViewById(R.id.tvInfo);

        red = new Paint();
        red.setStyle(Paint.Style.STROKE);
        red.setStrokeWidth(3);

        previewHolder = svPreview.getHolder();
        previewHolder.addCallback(this);
        previewHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
    }

    public void surfaceCreated(SurfaceHolder arg0) {
        camera = Camera.open();
        try {
            camera.setDisplayOrientation(90);
            camera.setFaceDetectionListener(this);
            camera.setPreviewDisplay(previewHolder);
        }
        catch (IOException e) {
            e.printStackTrace();
        }
    }

    public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
        // . . .
        camera.startPreview();
        camera.autoFocus(null);
        camera.startFaceDetection();
    }

    public void surfaceDestroyed(SurfaceHolder arg0) {
        camera.stopFaceDetection();
        camera.cancelAutoFocus();
        camera.stopPreview();
        camera.release();
        camera = null;
    }

    public void onFaceDetection(Face[] faces, Camera camera) {  
        tvInfo.setText("Faces: " + String.valueOf(faces.length));

        Canvas canvas = previewHolder.lockCanvas();
        for(int i=0; i < faces.length; i++) {
            Point leftEye = faces[i].leftEye;
            Point rightEye = faces[i].rightEye;
            // this is not working
            canvas.drawPoint(leftEye.x, leftEye.y, red);
        }
        previewHolder.unlockCanvasAndPost(canvas);
    }
}

With this code I keep getting this error:

09-03 19:35:42.743: E/SurfaceHolder(19394): Exception locking surface
09-03 19:35:42.743: E/SurfaceHolder(19394): java.lang.IllegalArgumentException
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.Surface.lockCanvasNative(Native Method)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.Surface.lockCanvas(Surface.java:76)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.SurfaceView$4.internalLockCanvas(SurfaceView.java:744)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.view.SurfaceView$4.lockCanvas(SurfaceView.java:720)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at com.bluetooth.activities.MyActivity.onFaceDetection(MyActivity.java:90)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.hardware.Camera$EventHandler.handleMessage(Camera.java:729)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.os.Handler.dispatchMessage(Handler.java:99)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.os.Looper.loop(Looper.java:137)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at android.app.ActivityThread.main(ActivityThread.java:4424)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at java.lang.reflect.Method.invokeNative(Native Method)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at java.lang.reflect.Method.invoke(Method.java:511)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
09-03 19:35:42.743: E/SurfaceHolder(19394):     at dalvik.system.NativeStart.main(Native Method)
09-03 19:35:42.743: W/dalvikvm(19394): threadid=1: thread exiting with uncaught exception (group=0x40a561f8)
09-03 19:35:42.766: E/AndroidRuntime(19394): FATAL EXCEPTION: main
09-03 19:35:42.766: E/AndroidRuntime(19394): java.lang.IllegalArgumentException
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.view.Surface.unlockCanvasAndPost(Native Method)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.view.SurfaceView$4.unlockCanvasAndPost(SurfaceView.java:775)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at com.bluetooth.activities.MyActivity.onFaceDetection(MyActivity.java:99)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.hardware.Camera$EventHandler.handleMessage(Camera.java:729)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.os.Handler.dispatchMessage(Handler.java:99)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.os.Looper.loop(Looper.java:137)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at android.app.ActivityThread.main(ActivityThread.java:4424)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at java.lang.reflect.Method.invokeNative(Native Method)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at java.lang.reflect.Method.invoke(Method.java:511)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
09-03 19:35:42.766: E/AndroidRuntime(19394):    at dalvik.system.NativeStart.main(Native Method)

Is there a problem with the camera trying to draw its preview on the same SurfaceView the face detection callback? How can I do this without layering SurfaceViews?

Mcnally

I added a bounty to this because from digging in the Android source it looks like the preview runs on a different thread than the rest of the app, which I believe means that your app can't get a Canvas object, or lock the Surface, because they are different threads. That said, I'm hoping someone who's dealt with this will respond – Prevot

How about using a FrameLayout on top of all these views with a transparent background! – Hemolysis

Answer 1

You can't lock and draw on a SurfaceView which has Type.PUSH_BUFFERS, (the one you're displaying frames to). You have to create another view above your original one in the Z direction and draw on a SurfaceView in that View.

So in your main.xml create a custom view below your original view in a FrameLayout.

Create and handle a SurfaceView in your Activity View. Add this view to the Camera preview display. Start your custom view passing the SurfaceHolder. In this view you can lock and draw on a canvas.

Buchmanism

Answer 2

As James pointed out you need to create custom surface which extends SurfaceView (I usually implement SurfaceHolder.Callback also):

public class CameraSurfacePreview extends SurfaceView implements SurfaceHolder.Callback

Constructor will be something like:

public CameraSurfacePreview(Context context) {
     super(context);
     ...
     mHolder = getHolder();
     mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
     ...

You need to bind camera with your surface after camera open call (if you implement implements SurfaceHolder.Callback put this somewhere inside overridden surfaceCreated):

mCamera = Camera.open();
mCamera.setPreviewDisplay(mHolder);

Finally you need to add instance of your custom surface somehere in activity content view:

CameraSurfacePreview cameraSurfacePreview = new CameraSurfacePreview(this);
//camera surface preview is first child!
((ViewGroup)findViewById(R.id.cameraLayout)).addView(cameraSurfacePreview, 0); 

In my example layout for activity looks something like(I am showing camera preview in main frame layout):

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android" 
              android:layout_width="fill_parent"        
                      android:layout_height="fill_parent" 
              android:layout_gravity="top|left"
              android:id="@+id/cameraLayout">

Plummy

Answer 3

I could said,

It's not about a Thread.

It not because of this line that make error either.

canvas.drawPoint(leftEye.x, leftEye.y, red);

It because of the canvas still using and can't lock it

If you carefully check, you will see this canvas you get is == null

canvas = canvasHolder.lockCanvas();
    if (canvas == null)  Log.i("Error", "Canvas == null!"); 

You might have question then where is it already use?

It already use to display to show what's going on to you! That is

camera.setPreviewDisplay(previewHolder);

So, I suggest, If you want to draw Point over your eye, you might need to have another SurfaceView / SurfaceHolder over you SurfaceView for preview camera :]

Name

Answer 4

The lockcanvas/unlockcanvasandpost approach is not appropriate when using openGL as the openGL code is controlling and locking the surface. If you want to use the standard 2d APIs, don't use OpenGL.

Kaylyn