Crop camera preview for TextureView

Question

I have a TextureView with a fixed width and height and I want to show a camera preview inside of it. I need to crop the camera preview so that it doesn't look stretched inside my TextureView. How to do the cropping? If I need to use OpenGL, how to tie the Surface Texture to OpenGL and how to do the cropping with OpenGL?

public class MyActivity extends Activity implements TextureView.SurfaceTextureListener 
{

   private Camera mCamera;
   private TextureView mTextureView;

  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_options);

    mTextureView = (TextureView) findViewById(R.id.camera_preview);
    mTextureView.setSurfaceTextureListener(this);
}

@Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
    mCamera = Camera.open();

    try 
    {
        mCamera.setPreviewTexture(surface);
        mCamera.startPreview();
    } catch (IOException ioe) {
        // Something bad happened
    }
}

@Override
public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
    mCamera.stopPreview();
    mCamera.release();
    return true;
}

@Override
public void onSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {
}

@Override
public void onSurfaceTextureUpdated(SurfaceTexture surface)
 {
    // Invoked every time there's a new Camera preview frame
 }
}

Also, after doing the preview correctly, I need to be able to read in real-time the pixels found in the center of the cropped image.

Tropopause

I am not sure it will help you bt you can try with less preview size.initially camera gives default preview size a larger one.. – Social

I know about preview sizes. This does not help me because the view that shows the camera preview has dimensions different from those available as preview sizes. – Tropopause

You can try with fragment on your screen. – Social

ok, to get the best preview size i agree with @Amrendra, by doing some math to get the closest preview size to the size of your view. and i think that is that closest thing that you can achieve – Shalandashale

Some devices have very few preview sizes to pick from and none are very close to the size I need. So this is not an option. – Tropopause

if you dont wanna process the image, you can put textureview inside linearlayout and add layout_gravity ="center" or center_vertically to your textrure view. here for more : #18051972 – Gallard

Answer 1

Provided earlier solution by @Romanski works fine but it scales with cropping. If you need to scale to fit, then use the following solution. Call updateTextureMatrix every time when surface view is changed: i.e. in onSurfaceTextureAvailable and in onSurfaceTextureSizeChanged methods. Also note that this solution relies that activity ignores configuration changes (i.e. android:configChanges="orientation|screenSize|keyboardHidden" or something like that):

private void updateTextureMatrix(int width, int height)
{
    boolean isPortrait = false;

    Display display = getWindowManager().getDefaultDisplay();
    if (display.getRotation() == Surface.ROTATION_0 || display.getRotation() == Surface.ROTATION_180) isPortrait = true;
    else if (display.getRotation() == Surface.ROTATION_90 || display.getRotation() == Surface.ROTATION_270) isPortrait = false;

    int previewWidth = orgPreviewWidth;
    int previewHeight = orgPreviewHeight;

    if (isPortrait)
    {
        previewWidth = orgPreviewHeight;
        previewHeight = orgPreviewWidth;
    }

    float ratioSurface = (float) width / height;
    float ratioPreview = (float) previewWidth / previewHeight;

    float scaleX;
    float scaleY;

    if (ratioSurface > ratioPreview)
    {
        scaleX = (float) height / previewHeight;
        scaleY = 1;
    }
    else
    {
        scaleX = 1;
        scaleY = (float) width / previewWidth;
    }

    Matrix matrix = new Matrix();

    matrix.setScale(scaleX, scaleY);
    textureView.setTransform(matrix);

    float scaledWidth = width * scaleX;
    float scaledHeight = height * scaleY;

    float dx = (width - scaledWidth) / 2;
    float dy = (height - scaledHeight) / 2;
    textureView.setTranslationX(dx);
    textureView.setTranslationY(dy);
}

Also you need the following fields:

private int orgPreviewWidth;
private int orgPreviewHeight;

initialize it in onSurfaceTextureAvailable mathod before calling updateTextureMatrix:

Camera.Parameters parameters = camera.getParameters();
parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);

Pair<Integer, Integer> size = getMaxSize(parameters.getSupportedPreviewSizes());
parameters.setPreviewSize(size.first, size.second);

orgPreviewWidth = size.first;
orgPreviewHeight = size.second;

camera.setParameters(parameters);

getMaxSize method:

private static Pair<Integer, Integer> getMaxSize(List<Camera.Size> list)
{
    int width = 0;
    int height = 0;

    for (Camera.Size size : list) {
        if (size.width * size.height > width * height)
        {
            width = size.width;
            height = size.height;
        }
    }

    return new Pair<Integer, Integer>(width, height);
}

And last thing - you need to correct camera rotation. So call setCameraDisplayOrientation method in Activity onConfigurationChanged method (and also make initial call in onSurfaceTextureAvailable method):

public static void setCameraDisplayOrientation(Activity activity, int cameraId, Camera camera)
{
    Camera.CameraInfo info = new Camera.CameraInfo();
    Camera.getCameraInfo(cameraId, info);
    int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();
    int degrees = 0;
    switch (rotation)
    {
        case Surface.ROTATION_0:
            degrees = 0;
            break;
        case Surface.ROTATION_90:
            degrees = 90;
            break;
        case Surface.ROTATION_180:
            degrees = 180;
            break;
        case Surface.ROTATION_270:
            degrees = 270;
            break;
    }

    int result;
    if (info.facing == Camera.CameraInfo.CAMERA_FACING_FRONT)
    {
        result = (info.orientation + degrees) % 360;
        result = (360 - result) % 360;  // compensate the mirror
    }
    else
    {  // back-facing
        result = (info.orientation - degrees + 360) % 360;
    }
    camera.setDisplayOrientation(result);

    Camera.Parameters params = camera.getParameters();
    params.setRotation(result);
    camera.setParameters(params);
}

Absenteeism

GIVE THIS MAN AN UPVOTE! thanks so much for this Ruslan. Ive bveen trying to attack this with surfaceviews, surfacetextures and a million other things for about a week now. I knew there should be some kind of view i could set a scaling matrix on but i just couldn't find it. this is perfect. I think there should be a link to this answer from the many other questions about camera preview cropping that are out there. – Branchiopod

edit: there is a slight logic issue in this code, which romanski seems to have solved, ill try and update this answer once i work out where the problem is – Branchiopod

For an example of doing this directly with SurfaceTexture and GLES, see the "texture from camera" activity in Grafika (github.com/google/grafika). The "zoom" action does a center crop and scale by manipulating texture coordinates. TextureView is doing something similar. The advantage of TextureView is that it's a whole lot easier to use. The "raw GLES" approach is much more complicated but much more flexible. – Lituus

Hi, thanks for useful post. This helped me to resolve the problem after which i have got another one: when moving the camera, the view is not smoothly. Seems like bad transformation or something like that. Can you help me please ? any ideas for such a behavior ? – Pleuron

I have a little trouble understanding this. I get error setParameters failed. I have followed this guide but I think and also make initial call in onSurfaceTextureAvailable method is where I'm getting a little confused. Is this supposed to be before I set Camera camera = Camera.open(); ? – Mandamus

Serious logic issue in the upper answer. Let suppose we have view 720 * 720 and camera preview is 1280 * 720. I got totally wrong scaling factor. – Uniform

Does this still work for api levels over 20? I am having issues with the translation on android devices from api level 21 – Hydrogenous

Answer 2

Just calculate the aspect ratio, generate a scaling matrix and apply it to the TextureView. Based on the aspect ratio of the surface and the aspect ratio of the preview image, the preview image is cropped on the top and the bottom or left and right. Another solution I found out is that if you open the camera before the SurfaceTexture is available, the preview is already scaled automatically. Just try to move mCamera = Camera.open(); to your onCreate function after you set the SurfaceTextureListener. This worked for me on the N4. With this solution you'll probably get problems when you rotate from portrait to landscape. If you need portrait and landscape support, then take the solution with the scale matrix!

private void initPreview(SurfaceTexture surface, int width, int height) {
    try {
        camera.setPreviewTexture(surface);
    } catch (Throwable t) {
        Log.e("CameraManager", "Exception in setPreviewTexture()", t);
    }

    Camera.Parameters parameters = camera.getParameters();
    previewSize = parameters.getSupportedPreviewSizes().get(0);

    float ratioSurface = width > height ? (float) width / height : (float) height / width;
    float ratioPreview = (float) previewSize.width / previewSize.height;

    int scaledHeight = 0;
    int scaledWidth = 0;
    float scaleX = 1f;
    float scaleY = 1f;

    boolean isPortrait = false;

    if (previewSize != null) {
        parameters.setPreviewSize(previewSize.width, previewSize.height);
        if (display.getRotation() == Surface.ROTATION_0 || display.getRotation() == Surface.ROTATION_180) {
            camera.setDisplayOrientation(display.getRotation() == Surface.ROTATION_0 ? 90 : 270);
            isPortrait = true;
        } else if (display.getRotation() == Surface.ROTATION_90 || display.getRotation() == Surface.ROTATION_270) {
            camera.setDisplayOrientation(display.getRotation() == Surface.ROTATION_90 ? 0 : 180);
            isPortrait = false;
        }
        if (isPortrait && ratioPreview > ratioSurface) {
            scaledWidth = width;
            scaledHeight = (int) (((float) previewSize.width / previewSize.height) * width);
            scaleX = 1f;
            scaleY = (float) scaledHeight / height;
        } else if (isPortrait && ratioPreview < ratioSurface) {
            scaledWidth = (int) (height / ((float) previewSize.width / previewSize.height));
            scaledHeight = height;
            scaleX = (float) scaledWidth / width;
            scaleY = 1f;
        } else if (!isPortrait && ratioPreview < ratioSurface) {
            scaledWidth = width;
            scaledHeight = (int) (width / ((float) previewSize.width / previewSize.height));
            scaleX = 1f;
            scaleY = (float) scaledHeight / height;
        } else if (!isPortrait && ratioPreview > ratioSurface) {
            scaledWidth = (int) (((float) previewSize.width / previewSize.height) * width);
            scaledHeight = height;
            scaleX = (float) scaledWidth / width;
            scaleY = 1f;
        }           
        camera.setParameters(parameters);
    }

    // calculate transformation matrix
    Matrix matrix = new Matrix();

    matrix.setScale(scaleX, scaleY);
    textureView.setTransform(matrix);
}

Steward

I can't get it to crop the preview. Are you sure it's possible to crop the preview inside the TextureView? – Tropopause

It's working for me that way. Basically it's not cropping but scaling. Camera preview is always fitted inside the TextureView - even if the ratio of the TextureView and the ratio of the camera preview are not the same. With the transform matrix you can set the scaling to compensate the distortion. – Steward

I know I'm quite late, but how would you adapt the rotation to the Camera2 API? There is no "setDisplayOrientation". I'm testing around with matrix.setRotate/postRotate/preRotate, but I never get the desired output. – Inlaid

On Camera2 I use another approach - calculate the matrix with setRectToRect (image size and surface size) and then based on the current display position I use postScale/postRotate on the matrix. I'm sorry, but I can't provide any code for that – Steward

Answer 3

I have just made a working app where I needed to show a preview with x2 of the actual input without getting a pixelated look. Ie. I needed to show the center part of a 1280x720 live preview in a 640x360 TextureView.

Here is what I did.

Set the camera preview to x2 resolution of what I needed:

params.setPreviewSize(1280, 720);

And then scale the texture view accordingly:

this.captureView.setScaleX(2f);
this.captureView.setScaleY(2f);

This runs without any hiccups on small devices.

Tommi

Thanks alot.You save me. – Mouflon

Answer 4

you could manipulate the byte[] data from onPreview().

I think you'll have to:

• put it in an Bitmap
• do the cropping in the Bitmap
• do a little stretching/resizing
• and pass the Bitmap to your SurfaceView

This is not a very performant way. Maybe you can manipulate the byte[] directly, but you will have to deal with picture formats like NV21.

Manganese

Answer 5

Answer given by @SatteliteSD is the most appropriate one. Every camera supports only certain preview sizes which is set at HAL. Hence if the available preview sizes doesn't suffice the requirement then you need to extract data from onPreview

Aciculate

Answer 6

For real-time manipulations of preview images of the camera, OpenCV for android is perfectly adapted. You'll find every sample needed here : http://opencv.org/platforms/android/opencv4android-samples.html, and as you'll see it works damn well in real time.

Disclaimer : setting up an opencv lib on android can be quite the headache depending on how you are experimented with c++/opencv/the ndk. In all cases, writing opencv code is never simple, but on the other hand it's a very powerful library.

Stidham

I've been looking at JavaCameraView myself and it seems to be doing the yuv->rgb conversion in native code one the CPU rather than on the gpu... i think this is leading to a real slowdown in the camera frames being delivered to the screen (delay and poor fps) – Branchiopod