Camera2 api Imageformat.yuv_420_888 results on rotated image

Question

Lots of questions have been made about camera2 api and RAW image format, but searching online I have still not found the answer (that's why I am here btw).

I am trying to do some real-time image processing on camera-captured frames using ImageReader and setRepeatingRequest with the front-facing camera. As suggested in some previous posts, I am acquiring the image in a RAW format (specifically Imageformat.yuv_420_888) in order to have a frame-rate around 30fps:

  imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 2); 

My image-processing algorithm requires an RGB image as input, so I need to convert from YUV to RGB. To do that I use ScriptIntrinsicYuvToRGB

  private static Bitmap YUV_420_888_toRGBIntrinsics(Image image) {

    if (image == null) return null;

    int W = image.getWidth();
    int H = image.getHeight();

    Image.Plane Y = image.getPlanes()[0];
    Image.Plane U = image.getPlanes()[1];
    Image.Plane V = image.getPlanes()[2];

    int Yb = Y.getBuffer().remaining();
    int Ub = U.getBuffer().remaining();
    int Vb = V.getBuffer().remaining();

    byte[] data = new byte[Yb + Ub + Vb];


    Y.getBuffer().get(data, 0, Yb);
    V.getBuffer().get(data, Yb, Vb);
    U.getBuffer().get(data, Yb + Vb, Ub);



    rs = RenderScript.create(context);
    ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

    Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(data.length);
    Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);

    Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(W).setY(H);
    Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);


    final Bitmap bmpout = Bitmap.createBitmap(W, H, Bitmap.Config.ARGB_8888); 

    in.copyFromUnchecked(data);

    yuvToRgbIntrinsic.setInput(in);
    yuvToRgbIntrinsic.forEach(out);
    out.copyTo(bmpout);
    image.close();
    return bmpout ;
}

This method is quite fast since I can convert a 1080p image in less than 20ms. The only issue is that the image result is rotated by 270 degrees (i.e. picture is taken in landscape mode). Even if I set JPEG_ORIENTATION in the camera builder settings,captureRequestBuilder.set(CaptureRequest.JPEG_ORIENTATION, characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)); the result is still the same.

Here my question:

• Is there a way to get back a rotated image through renderscript intrinsics?
• Is there a "rotate" function that does not allocate memory ?
• Are there settings for image rotation of a YUV type? Other the solutions that I have tried - Matrix rotation, YUV array rotation - are quite slow. Moreover I think that rotating an image by 90/180/270 is an easy task if done after having taken it, just need to save rows instead of columns (somehow).

Sforza

Hi, did you resolved the issue? can you post the resulting code if yes? thank you. – Adherent

Answer 1

No, there's no built-in rotation for YUV output. To minimize overhead, it's always produced as-is from the image sensor. You can read the SENSOR_ORIENTATION field to determine how the image sensor is placed on the device; typically the long edge of the image sensor lines up with the long edge of the Android device, but that still leaves two rotations that are valid.

Also, if your goal is to have the image 'upright', then you also need to read what the device's orientation is from the accelerometer, and add that in to the rotation.

You're doing a copy already getting the frame from the Image into the Allocation, so doing a 90/180/270 degree rotation then is relatively straightforward, though memory-bandwidth-intensive.

You can also take a look at one of Google's sample apps, HdrViewfinderDemo, which pipes camera data into RenderScript without the intermediate copy you're doing, and then converts to RGB to draw to a SurfaceView. It doesn't have a rotation in it now, but you could adjust the lookup done via rsGetElementAtYuv_uchar_* to do 90-increments.

Meteorograph

hello, what do you mean with "It doesn't have a rotation in it now, but you could adjust the lookup done via rsGetElementAtYuv_uchar_* to do 90-increments". I try to take the HdrViewfinderDemo approach to process camera data using Renderscript, but I could not find out how to make the output in portrait mode. The output is always in landscape mode. – Shurlock

YUV output is always the same orientation. If you ask for WxH size, you get a WxH YUV buffer no matter how you're holding the phone or the UI is oriented. If you need to rotate the buffer for display in a portrait ImageView or similar, you need to do it yourself. – Meteorograph

@EddyTalvala: How can I rotate the buffer ? – Daffy

@EddyTalvala: can you pls take a look at #57725427 and could you help me out with this ? – Daffy

Answer 2

A few days of research confirmed to me Eddy's answer: there is no built-in way of rotating the YUV Image planes, however, I did not find implementing the U and V rotations at all straight-forward for semi-planar format.

For one, the U and V ByteBuffers returned in a semi-planar YUV 4:2:0 Image acquired from ImageReader.acquireNextImage() have equivalent/over-lapping "U1,V1,U2,V2...Un,Vn" values for all indices except that the final value of the U buffer does not contain the last V value, Vn, and the V buffer does not contain the first U value, U1.

I did not see the above detail anywhere in Android docs, but luckily I came across this StackOverflow response which explained: "the ByteBuffers for the two planes overlap, with the second chroma plane's buffer starting one byte ahead of the first chroma plane's buffer."

So, Planes[1] (the U buffer) and Planes[2] (the V buffer) have the same size of ImageWidth * ImageHeight / 2 - 1 which is also equivalent to Y.length/2 - 1.

Mr. GPT and online sources all lack implementations with this in mind, so I made a function that takes a YUV semi-planar Image and uses its ByteBuffer[] of YUV values and outputs a 90°rotated ByteBuffer[], without any conversion necessary into a byte[] in between. Here is the Gist for it:

https://gist.github.com/stefankiesz/9604bd835364a3a9deaf1482424c3e24

Skilken