Android: how to display camera preview with callback?

Question

What I need to do is quite simple, I want to manually display preview from camera using camera callback and I want to get at least 15fps on a real device. I don't even need the colors, I just need to preview grayscale image. Images from camera are in YUV format and you have to process it somehow, which is the main performance problem. I'm using API 8.

In all cases I'm using camera.setPreviewCallbackWithBuffer(), that is faster than camera.setPreviewCallback(). It seems that I cant get about 24 fps here, if I'm not displaying the preview. So there is not the problem.

I have tried these solutions:

1. Display camera preview on a SurfaceView as a Bitmap. It works, but the performance is about 6fps.

baos = new ByteOutputStream();
yuvimage=new YuvImage(cameraFrame, ImageFormat.NV21, prevX, prevY, null);

yuvimage.compressToJpeg(new Rect(0, 0, prevX, prevY), 80, baos);
jdata = baos.toByteArray();

bmp = BitmapFactory.decodeByteArray(jdata, 0, jdata.length); // Convert to Bitmap, this is the main issue, it takes a lot of time

canvas.drawBitmap(bmp , 0, 0, paint);

---

2. Display camera preview on a GLSurfaceView as a texture. Here I was displaying only luminance data (greyscale image), which is quite easy, it requires only one arraycopy() on each frame. I can get about 12fps, but I need to apply some filters to the preview and it seems, that it can't be done fast in OpenGL ES 1. So I can't use this solution. Some details of this in another question.

---

3. Display camera preview on a (GL)SurfaceView using NDK to process the YUV data. I find a solution here that uses some C function and NDK. But I didn't manage to use it, here some more details. But anyway, this solution is done to return ByteBuffer to display it as a texture in OpenGL and it won't be faster than the previous attempt. So I would have to modify it to return int[] array, that can be drawn with canvas.drawBitmap(), but I don't understand C enough to do this.

---

So, is there any other way that I'm missing or some improvement to the attempts I tried?

Endgame

Answer 1

I'm working on exactly the same issue, but haven't got quite as far as you have.

Have you considered drawing the pixels directly to the canvas without encoding them to JPEG first? Inside the OpenCV kit http://sourceforge.net/projects/opencvlibrary/files/opencv-android/2.3.1/OpenCV-2.3.1-android-bin.tar.bz2/download (which doesn't actually use opencv; don't worry), there's a project called tutorial-0-androidcamera that demonstrates converting the YUV pixels to RGB and then writing them directly to a bitmap.

The relevant code is essentially:

public void onPreviewFrame(byte[] data, Camera camera, int width, int height) {
    int frameSize = width*height;
    int[] rgba = new int[frameSize+1];

    // Convert YUV to RGB
    for (int i = 0; i < height; i++)
        for (int j = 0; j < width; j++) {
            int y = (0xff & ((int) data[i * width + j]));
            int u = (0xff & ((int) data[frameSize + (i >> 1) * width + (j & ~1) + 0]));
            int v = (0xff & ((int) data[frameSize + (i >> 1) * width + (j & ~1) + 1]));
            y = y < 16 ? 16 : y;

            int r = Math.round(1.164f * (y - 16) + 1.596f * (v - 128));
            int g = Math.round(1.164f * (y - 16) - 0.813f * (v - 128) - 0.391f * (u - 128));
            int b = Math.round(1.164f * (y - 16) + 2.018f * (u - 128));

            r = r < 0 ? 0 : (r > 255 ? 255 : r);
            g = g < 0 ? 0 : (g > 255 ? 255 : g);
            b = b < 0 ? 0 : (b > 255 ? 255 : b);

            rgba[i * width + j] = 0xff000000 + (b << 16) + (g << 8) + r;
        }

    Bitmap bmp = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
    bmp.setPixels(rgba, 0/* offset */, width /* stride */, 0, 0, width, height);
    Canvas canvas = mHolder.lockCanvas();
    if (canvas != null) {
        canvas.drawBitmap(bmp, (canvas.getWidth() - width) / 2, (canvas.getHeight() - height) / 2, null);
        mHolder.unlockCanvasAndPost(canvas);
    } else {
        Log.w(TAG, "Canvas is null!");
    }
    bmp.recycle();
}

Of course you'd have to adapt it to meet your needs (ex. not allocating rgba each frame), but it might be a start. I'd love to see if it works for you or not -- i'm still fighting problems orthogonal to yours at the moment.

Strunk

I will try this tomorrow, but I'm quite sure that there will be no performance improvement, because this works more or less the same as the version with YuvImage and BitmapFactory classes. But I'll give it a try and we'll see. Thanks. – Endgame

So, as I expected, it has the worst performance from all the cases, even when I tried to optimize it a bit... – Endgame

I am getting java.lang.ArrayIndexOutOfBoundsException at this line int u = (0xff & ((int) data[frameSize + (i >> 1) * width + (j & ~1) + 0])); – Subside

Answer 2

I think Michael's on the right track. First you can try this method to convert from RGB to Grayscale. Clearly it's doing almost the same thing as his,but a little more succinctly for what you want.

//YUV Space to Greyscale
static public void YUVtoGrayScale(int[] rgb, byte[] yuv420sp, int width, int height){
    final int frameSize = width * height;
    for (int pix = 0; pix < frameSize; pix++){
        int pixVal = (0xff & ((int) yuv420sp[pix])) - 16;
        if (pixVal < 0) pixVal = 0;
        if (pixVal > 255) pixVal = 255;
        rgb[pix] = 0xff000000 | (pixVal << 16) | (pixVal << 8) | pixVal;
    }
}

}

Second, don't create a ton of work for the garbage collector. Your bitmaps and arrays are going to be a fixed size. Create them once, not in onFramePreview.

Doing that you'll end up with something that looks like this:

    public PreviewCallback callback = new PreviewCallback() {
    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        if ( (mSelectView == null) || !inPreview )
            return;
        if (mSelectView.mBitmap == null)
        {
            //initialize SelectView bitmaps, arrays, etc
            //mSelectView.mBitmap = Bitmap.createBitmap(mSelectView.mImageWidth, mSelectView.mImageHeight, Bitmap.Config.RGB_565);
           //etc

        }
        //Pass Image Data to SelectView
        System.arraycopy(data, 0, mSelectView.mYUVData, 0, data.length);
        mSelectView.invalidate();
    }
};

And then the canvas where you want to put it looks like this:

class SelectView extends View {
Bitmap mBitmap;
Bitmap croppedView;
byte[] mYUVData;
int[] mRGBData;
int mImageHeight;
int mImageWidth;

public SelectView(Context context){
    super(context);
    mBitmap = null;
    croppedView = null;
}

@Override
protected void onDraw(Canvas canvas){
    if (mBitmap != null)
    {
        int canvasWidth = canvas.getWidth();
        int canvasHeight = canvas.getHeight();
        // Convert from YUV to Greyscale
        YUVtoGrayScale(mRGBData, mYUVData, mImageWidth, mImageHeight);
            mBitmap.setPixels(mRGBData, 0, mImageWidth, 0, 0, mImageWidth, mImageHeight);
            Rect crop = new Rect(180, 220, 290, 400);
        Rect dst = new Rect(0, 0, canvasWidth, (int)(canvasHeight/2));
        canvas.drawBitmap(mBitmap, crop, dst, null);
    }
    super.onDraw(canvas);
}

This example shows a cropped and distorted selection of the camera preview in real time, but you get the idea. It runs at high FPS on a Nexus S in greyscale and should work for your needs as well.

Vonvona

Second, don't create a ton of work for the garbage collector.. Your onDraw tell us otherwise. – Rorke

Answer 3

Is this not what you want? Just use a SurfaceView in your layout, then somewhere in your init like onResume():

SurfaceView surfaceView = ...
SurfaceHolder holder = surfaceView.getHolder();
...
Camera camera = ...;
camera.setPreviewDisplay(holder);

It just sends the frames straight to the view as fast as they arrive.

If you want grayscale, modify the camera parameters with setColorEffect("mono").

Allantois

Thanks, but I really need to use callback and display images manually. I don't need only grayscale image, I'm using some more filters, that I didn't mention in my question... – Endgame

Answer 4

For very basic and simple effects, there is

Camera.Parameters parameters = mCamera.getParameters();
parameters.setColorEffect(Parameters.EFFECT_AQUA);

I figured out that this effects do DIFFERENTLY depending on the device. For instance, on my phone (galaxy s II) it looks kinda like a comic effect as in contrast to the galaxy s 1 it is 'just' a blue shade.

It's pro: It's working as live-preview.

I looked around some other camera apps and they obviously also faced this problem. So what did they do? They are capturing the default camera image, applying a filter to the bitmap data, and show this image in a simple ImageView. It's for sure not that cool as in live preview, but you won't ever face performance problems.

Outlawry

Thanks, I know about these camera effects, but I wanted to do some contrast adjustement, color substitution etc. But you're right, it seems that there is really not a good working solution to live preview, so I'll have to do something like you described... :/ – Endgame

Answer 5

I believe I read in a blog that the grayscale data is in the first x*y bytes. Yuv should represent luminance, so the data is there, although it isn't a perfect grayscale. Its great for relative brightness, but not grayscale, as each color isn't as bright as each other in rgb. Green is usually given a stronger weight in luminosity conversions. Hope this helps!

Seminar

Answer 6

Is there any special reason that you are forced to use GLES 1.0 ?

Because if not, see the accepted answer here: Android SDK: Get raw preview camera image without displaying it

Generally it mentions using Camera.setPreviewTexture() in combination with GLES 2.0. In GLES 2.0 you can render a full-screen-quad all over the screen, and create whatever effect you want.

It's most likely the fastest way possible.

Vitrification

Example: github.com/google/grafika/blob/master/src/com/android/grafika/… . It currently has a shader that puts a B&W image in the red color channel (search for "rosyFragment"). Assigning the same value to R/G/B would yield a B&W image. – Embolectomy