How can I manipulate the camera preview?

Question

There are several tutorials out there which explain how to get a simple camera preview up and running on an android device. But i couldn't find any example which explains how to manipulate the image before it's being rendered.
What I want to do is implementing custom color filters to simulate e.g. red and/or green deficiency.

Felix

if u could explain with code, what ur trying to do, it would be easy to help u – Outwear

github.com/whiskeysierra/impaired-vision/blob/master/src/org/… This is what i currently have. What i would like to do is registering some kind of callback (Camera.PreviewCallback?) which i then use to manipulate the current frame. – Felix

The link doesn't work anymore, but the repo still exists, in case anybody is still interested: github.com/whiskeysierra/impaired-vision – Felix

did you try using GPUImage, like i mentioned in my answer ? – Thyself

Answer 1

I did some research on this and put together a working(ish) example. Here's what I found. It's pretty easy to get the raw data coming off of the camera. It's returned as a YUV byte array. You'd need to draw it manually onto a surface in order to be able to modify it. To do that you'd need to have a SurfaceView that you can manually run draw calls with. There are a couple of flags you can set that accomplish that.

In order to do the draw call manually you'd need to convert the byte array into a bitmap of some sort. Bitmaps and the BitmapDecoder don't seem to handle the YUV byte array very well at this point. There's been a bug filed for this but don't don't what the status is on that. So people have been trying to decode the byte array into an RGB format themselves.

Seems like doing the decoding manually has been kinda slow and people have had various degrees of success with it. Something like this should probably really be done with native code at the NDK level.

Still, it is possible to get it working. Also, my little demo is just me spending a couple of hours hacking stuff together (I guess doing this caught my imagination a little too much ;)). So chances are with some tweaking you could much improve what I've managed to get working.

This little code snip contains a couple of other gems I found as well. If all you want is to be able to draw over the surface you can override the surface's onDraw function - you could potentially analyze the returned camera image and draw an overlay - it'd be much faster than trying to process every frame. Also, I changed the SurfaceHolder.SURFACE_TYPE_NORMAL from what would be needed if you wanted a camera preview to show up. So a couple of changes to the code - the commented out code:

//try { mCamera.setPreviewDisplay(holder); } catch (IOException e)
//  { Log.e("Camera", "mCamera.setPreviewDisplay(holder);"); }

And the:

SurfaceHolder.SURFACE_TYPE_NORMAL //SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS - for preview to work

Should allow you to overlay frames based on the camera preview on top of the real preview.

Anyway, here's a working piece of code - Should give you something to start with.

Just put a line of code in one of your views like this:

<pathtocustomview.MySurfaceView android:id="@+id/surface_camera"
    android:layout_width="fill_parent" android:layout_height="10dip"
    android:layout_weight="1">
</pathtocustomview.MySurfaceView>

And include this class in your source somewhere:

package pathtocustomview;

import java.io.IOException;
import java.nio.Buffer;

import android.content.Context;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Canvas;
import android.graphics.Paint;
import android.graphics.Rect;
import android.hardware.Camera;
import android.util.AttributeSet;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceHolder.Callback;
import android.view.SurfaceView;

public class MySurfaceView extends SurfaceView implements Callback,
    Camera.PreviewCallback {

    private SurfaceHolder mHolder;

    private Camera mCamera;
    private boolean isPreviewRunning = false;
    private byte [] rgbbuffer = new byte[256 * 256];
    private int [] rgbints = new int[256 * 256];

    protected final Paint rectanglePaint = new Paint();

    public MySurfaceView(Context context, AttributeSet attrs) {
    super(context, attrs);
        rectanglePaint.setARGB(100, 200, 0, 0);
        rectanglePaint.setStyle(Paint.Style.FILL);
        rectanglePaint.setStrokeWidth(2);

        mHolder = getHolder();
        mHolder.addCallback(this);
        mHolder.setType(SurfaceHolder.SURFACE_TYPE_NORMAL);
    }

    @Override
    protected void onDraw(Canvas canvas) {
        canvas.drawRect(new Rect((int) Math.random() * 100,
            (int) Math.random() * 100, 200, 200), rectanglePaint);
        Log.w(this.getClass().getName(), "On Draw Called");
    }

    public void surfaceChanged(SurfaceHolder holder, int format, int width,
            int height) {
    }

    public void surfaceCreated(SurfaceHolder holder) {
        synchronized (this) {
            this.setWillNotDraw(false); // This allows us to make our own draw
                                    // calls to this canvas

            mCamera = Camera.open();

            Camera.Parameters p = mCamera.getParameters();
            p.setPreviewSize(240, 160);
            mCamera.setParameters(p);


            //try { mCamera.setPreviewDisplay(holder); } catch (IOException e)
            //  { Log.e("Camera", "mCamera.setPreviewDisplay(holder);"); }

            mCamera.startPreview();
            mCamera.setPreviewCallback(this);

        }
    }

    public void surfaceDestroyed(SurfaceHolder holder) {
        synchronized (this) {
            try {
                if (mCamera != null) {
                    mCamera.stopPreview();
                    isPreviewRunning = false;
                    mCamera.release();
                }
            } catch (Exception e) {
                Log.e("Camera", e.getMessage());
            }
        }
    }

    public void onPreviewFrame(byte[] data, Camera camera) {
        Log.d("Camera", "Got a camera frame");

        Canvas c = null;

        if(mHolder == null){
            return;
        }

        try {
            synchronized (mHolder) {
                c = mHolder.lockCanvas(null);

                // Do your drawing here
                // So this data value you're getting back is formatted in YUV format and you can't do much
                // with it until you convert it to rgb
                int bwCounter=0;
                int yuvsCounter=0;
                for (int y=0;y<160;y++) {
                    System.arraycopy(data, yuvsCounter, rgbbuffer, bwCounter, 240);
                    yuvsCounter=yuvsCounter+240;
                    bwCounter=bwCounter+256;
                }

                for(int i = 0; i < rgbints.length; i++){
                    rgbints[i] = (int)rgbbuffer[i];
                }

                //decodeYUV(rgbbuffer, data, 100, 100);
                c.drawBitmap(rgbints, 0, 256, 0, 0, 256, 256, false, new Paint());

                Log.d("SOMETHING", "Got Bitmap");

            }
        } finally {
            // do this in a finally so that if an exception is thrown
            // during the above, we don't leave the Surface in an
            // inconsistent state
            if (c != null) {
                mHolder.unlockCanvasAndPost(c);
            }
        }
    }
}

Tanberg

a got only errors while I'm trying your method, do you have maybe full working example? Thanks – Thud

c = mHolder.lockCanvas(null); c is null 08-05 18:49:23.419: E/SurfaceHolder(13927): Exception locking surface 08-05 18:49:23.419: E/SurfaceHolder(13927): java.lang.IllegalArgumentException 08-05 18:49:23.419: E/SurfaceHolder(13927): at android.view.Surface.nativeLockCanvas(Native Method) 08-05 18:49:23.419: E/SurfaceHolder(13927): at android.view.Surface.lockCanvas(Surface.java:243) 08-05 18:49:23.419: E/SurfaceHolder(13927): at android.view.SurfaceView$4.internalLockCanvas(SurfaceView.java:814) Please help – Muhammadan

where do you call onPreviewFrame? And what is "data"? – Wellfixed

It looks like the numbers 160 and 240 in the for-loop are the image size, but why 256 on "bwCounter=bwCounter+256" and in the call to drawBitmap? In drawBitmap some of the 256 surely should have been width or height, right? – Rhea

this is a difficult topic to search, the terms return so many results that are not relevant. FX, Live Fx, frames .. i had all but given up and then found this. I am not an app developer, but I desperately need this tool for adding logos to images pre-capture. (for sizing and perspective) .. to be stack compliant I'll try my best to phrase as a question. It there a working example or an app that demonstrates the layer/mask frame feature the code above? – Trapezium

Answer 2

I used walta's solution but I had some problems with YUV conversion, camera frames output sizes and crash on camera release.

Finally the following code worked for me:

public class MySurfaceView extends SurfaceView implements Callback, Camera.PreviewCallback {

private static final String TAG = "MySurfaceView";

private int width;
private int height;

private SurfaceHolder mHolder;

private Camera mCamera;
private int[] rgbints;

private boolean isPreviewRunning = false; 

private int mMultiplyColor;

public MySurfaceView(Context context, AttributeSet attrs) {
    super(context, attrs);

    mHolder = getHolder();
    mHolder.addCallback(this);
    mMultiplyColor = getResources().getColor(R.color.multiply_color);
}

// @Override
// protected void onDraw(Canvas canvas) {
// Log.w(this.getClass().getName(), "On Draw Called");
// }

@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

}

@Override
public void surfaceCreated(SurfaceHolder holder) {
    synchronized (this) {
        if (isPreviewRunning)
            return;

        this.setWillNotDraw(false); // This allows us to make our own draw calls to this canvas


        mCamera = Camera.open();
        isPreviewRunning = true;
        Camera.Parameters p = mCamera.getParameters();
        Size size = p.getPreviewSize();
        width = size.width;
        height = size.height;
        p.setPreviewFormat(ImageFormat.NV21);
        showSupportedCameraFormats(p);
        mCamera.setParameters(p);

        rgbints = new int[width * height];

        // try { mCamera.setPreviewDisplay(holder); } catch (IOException e)
        // { Log.e("Camera", "mCamera.setPreviewDisplay(holder);"); }

        mCamera.startPreview();
        mCamera.setPreviewCallback(this);

    }
}


@Override
public void surfaceDestroyed(SurfaceHolder holder) {
    synchronized (this) {
        try {
            if (mCamera != null) {
                //mHolder.removeCallback(this);
                mCamera.setPreviewCallback(null);
                mCamera.stopPreview();
                isPreviewRunning  = false;
                mCamera.release();
            }
        } catch (Exception e) {
            Log.e("Camera", e.getMessage());
        }
    }
}

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
    // Log.d("Camera", "Got a camera frame");
    if (!isPreviewRunning)
        return;

    Canvas canvas = null;

    if (mHolder == null) {
        return;
    }

    try {
        synchronized (mHolder) {
            canvas = mHolder.lockCanvas(null);
            int canvasWidth = canvas.getWidth();
            int canvasHeight = canvas.getHeight();

            decodeYUV(rgbints, data, width, height);

            // draw the decoded image, centered on canvas
            canvas.drawBitmap(rgbints, 0, width, canvasWidth-((width+canvasWidth)>>1), canvasHeight-((height+canvasHeight)>>1), width, height, false, null);

            // use some color filter
            canvas.drawColor(mMultiplyColor, Mode.MULTIPLY);

        }
    }  catch (Exception e){
        e.printStackTrace();
    } finally {
        // do this in a finally so that if an exception is thrown
        // during the above, we don't leave the Surface in an
        // inconsistent state
        if (canvas != null) {
            mHolder.unlockCanvasAndPost(canvas);
        }
    }
}



/**
 * Decodes YUV frame to a buffer which can be use to create a bitmap. use
 * this for OS < FROYO which has a native YUV decoder decode Y, U, and V
 * values on the YUV 420 buffer described as YCbCr_422_SP by Android
 * 
 * @param rgb
 *            the outgoing array of RGB bytes
 * @param fg
 *            the incoming frame bytes
 * @param width
 *            of source frame
 * @param height
 *            of source frame
 * @throws NullPointerException
 * @throws IllegalArgumentException
 */
public void decodeYUV(int[] out, byte[] fg, int width, int height) throws NullPointerException, IllegalArgumentException {
    int sz = width * height;
    if (out == null)
        throw new NullPointerException("buffer out is null");
    if (out.length < sz)
        throw new IllegalArgumentException("buffer out size " + out.length + " < minimum " + sz);
    if (fg == null)
        throw new NullPointerException("buffer 'fg' is null");
    if (fg.length < sz)
        throw new IllegalArgumentException("buffer fg size " + fg.length + " < minimum " + sz * 3 / 2);
    int i, j;
    int Y, Cr = 0, Cb = 0;
    for (j = 0; j < height; j++) {
        int pixPtr = j * width;
        final int jDiv2 = j >> 1;
    for (i = 0; i < width; i++) {
        Y = fg[pixPtr];
        if (Y < 0)
            Y += 255;
        if ((i & 0x1) != 1) {
            final int cOff = sz + jDiv2 * width + (i >> 1) * 2;
            Cb = fg[cOff];
            if (Cb < 0)
                Cb += 127;
            else
                Cb -= 128;
            Cr = fg[cOff + 1];
            if (Cr < 0)
                Cr += 127;
            else
                Cr -= 128;
        }
        int R = Y + Cr + (Cr >> 2) + (Cr >> 3) + (Cr >> 5);
        if (R < 0)
            R = 0;
        else if (R > 255)
            R = 255;
        int G = Y - (Cb >> 2) + (Cb >> 4) + (Cb >> 5) - (Cr >> 1) + (Cr >> 3) + (Cr >> 4) + (Cr >> 5);
        if (G < 0)
            G = 0;
        else if (G > 255)
            G = 255;
        int B = Y + Cb + (Cb >> 1) + (Cb >> 2) + (Cb >> 6);
        if (B < 0)
            B = 0;
        else if (B > 255)
            B = 255;
        out[pixPtr++] = 0xff000000 + (B << 16) + (G << 8) + R;
    }
    }

}

private void showSupportedCameraFormats(Parameters p) {
    List<Integer> supportedPictureFormats = p.getSupportedPreviewFormats();
    Log.d(TAG, "preview format:" + cameraFormatIntToString(p.getPreviewFormat()));
    for (Integer x : supportedPictureFormats) {
        Log.d(TAG, "suppoterd format: " + cameraFormatIntToString(x.intValue()));
    }

}

private String cameraFormatIntToString(int format) {
    switch (format) {
    case PixelFormat.JPEG:
        return "JPEG";
    case PixelFormat.YCbCr_420_SP:
        return "NV21";
    case PixelFormat.YCbCr_422_I:
        return "YUY2";
    case PixelFormat.YCbCr_422_SP:
        return "NV16";
    case PixelFormat.RGB_565:
        return "RGB_565";
    default:
        return "Unknown:" + format;

        }
    }
}

To Use it, run from you activity's onCreate the following code:

            SurfaceView surfaceView = new MySurfaceView(this, null);
        RelativeLayout.LayoutParams layoutParams = new RelativeLayout.LayoutParams(RelativeLayout.LayoutParams.MATCH_PARENT, RelativeLayout.LayoutParams.MATCH_PARENT);
        surfaceView.setLayoutParams(layoutParams);
        mRelativeLayout.addView(surfaceView);

Depside

I tried different solutions presented in this thread and my own as well. I get the below exception.I dont think one can lock the canvas from onPreviewFrame method atleast from Android 4.4 onward E/SurfaceHolder( 2872): java.lang.IllegalArgumentException E/SurfaceHolder( 2872): at android.view.Surface.nativeLockCanvas(Native Method) E/SurfaceHolder( 2872): at android.view.Surface.lockCanvas(Surface.java:243) – Sheer

I implemented the same solution but in my case I am getting a black surfaceview without any exceptions. – Jaborandi

I just see black. There is no video stream from the camera. I did everything like you did. E/AwareLog: AtomicFileUtils: readFileLines file not exist: android.util.AtomicFile@163047 - E/MemoryLeakMonitorManager: MemoryLeakMonitor.jar is not exist! - E/SpannableStringBuilder: SPAN_EXCLUSIVE_EXCLUSIVE spans cannot have a zero length – Saluki

Answer 3

Have you looked at GPUImage ?

It was originally an OSX/iOS library made by Brad Larson, that exists as an Objective-C wrapper around OpenGL/ES.

https://github.com/BradLarson/GPUImage

The people at CyberAgent have made an Android port (which doesn't have complete feature parity), which is a set of Java wrappers on top of the OpenGLES stuff. It's relatively high level, and pretty easy to implement, with a lot of the same functionality mentioned above...

https://github.com/CyberAgent/android-gpuimage

Thyself