Convert Android camera2 api YUV_420_888 to RGB

Question

I am writing an app that takes the camera feed, converts it to rgb, in order to do some processing.

It works fine on the old camera implementation which uses NV21 Yuv format. The issue I am having is with the new Yuv format, YUV_420_888. The image is no longer converted correctly to RGB in the new Camera2 Api which sends YUV_420_888 yuv format instead of NV21 (YUV_420_SP) format.

Can someone please tell me how should I convert YUV_420_888 to RGB?

Thanks

Uranalysis

That is for the older camera implementation, that doesn't help me. Thank you though. – Uranalysis

Has anyone converted YUV_420_888 to NV21 ( YUV_420_SP)? – Saccharo

@ConstantinGeorgiu, have you solved above problem. – Asmara

My solution takes the media.image as input and returns the Bitmap https://mcmap.net/q/298568/-has-anyone-managed-to-obtain-a-yuv_420_888-frame-using-renderscript-and-the-new-camera-api). – Saad

Have a look on this link I use this method to solve my problem. – Pantaloon

Answer 1

Camera2 YUV_420_888 to RGB Mat(opencv) in Java

@Override
    public void onImageAvailable(ImageReader reader){
        Image image = null;

        try {
            image = reader.acquireLatestImage();
            if (image != null) {

                byte[] nv21;
                ByteBuffer yBuffer = mImage.getPlanes()[0].getBuffer();
                ByteBuffer uBuffer = mImage.getPlanes()[1].getBuffer();
                ByteBuffer vBuffer = mImage.getPlanes()[2].getBuffer();

                int ySize = yBuffer.remaining();
                int uSize = uBuffer.remaining();
                int vSize = vBuffer.remaining();

                nv21 = new byte[ySize + uSize + vSize];

                //U and V are swapped
                yBuffer.get(nv21, 0, ySize);
                vBuffer.get(nv21, ySize, vSize);
                uBuffer.get(nv21, ySize + vSize, uSize);

                Mat mRGB = getYUV2Mat(nv21);



            }
        } catch (Exception e) {
            Log.w(TAG, e.getMessage());
        }finally{
            image.close();// don't forget to close
        }
    }



    public Mat getYUV2Mat(byte[] data) {
    Mat mYuv = new Mat(image.getHeight() + image.getHeight() / 2, image.getWidth(), CV_8UC1);
    mYuv.put(0, 0, data);
    Mat mRGB = new Mat();
    cvtColor(mYuv, mRGB, Imgproc.COLOR_YUV2RGB_NV21, 3);
    return mRGB;
}

Donatus

This is the fastest method I've found. Other conversions have for loops, which slows down a real time processing processing scenario. – Wyly

Note that this only works when the underlying buffers are NV21, i.e. when the uBuffer and vBuffer overlap, see https://mcmap.net/q/429273/-camera2-captured-picture-conversion-from-yuv_420_888-to-nv21. – Platonism

Worked nicely for converting Android CameraX Image Analysis frames (YUV) into RBG, thanks. – Reminiscence

Answer 2

In my approach I use OpenCV Mat and script from https://gist.github.com/camdenfullmer/dfd83dfb0973663a7974

First of all you convert your YUV_420_888 Image to Mat with the code in the link above.

*mImage is my Image object which i get in ImageReader.OnImageAvailableListener

Mat mYuvMat = imageToMat(mImage);

public static Mat imageToMat(Image image) {
    ByteBuffer buffer;
    int rowStride;
    int pixelStride;
    int width = image.getWidth();
    int height = image.getHeight();
    int offset = 0;

    Image.Plane[] planes = image.getPlanes();
    byte[] data = new byte[image.getWidth() * image.getHeight() * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
    byte[] rowData = new byte[planes[0].getRowStride()];

    for (int i = 0; i < planes.length; i++) {
        buffer = planes[i].getBuffer();
        rowStride = planes[i].getRowStride();
        pixelStride = planes[i].getPixelStride();
        int w = (i == 0) ? width : width / 2;
        int h = (i == 0) ? height : height / 2;
        for (int row = 0; row < h; row++) {
            int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
            if (pixelStride == bytesPerPixel) {
                int length = w * bytesPerPixel;
                buffer.get(data, offset, length);

                if (h - row != 1) {
                    buffer.position(buffer.position() + rowStride - length);
                }
                offset += length;
            } else {


                if (h - row == 1) {
                    buffer.get(rowData, 0, width - pixelStride + 1);
                } else {
                    buffer.get(rowData, 0, rowStride);
                }

                for (int col = 0; col < w; col++) {
                    data[offset++] = rowData[col * pixelStride];
                }
            }
        }
    }

    Mat mat = new Mat(height + height / 2, width, CvType.CV_8UC1);
    mat.put(0, 0, data);

    return mat;
}

We have 1 channel YUV Mat. Define new Mat for BGR(not RGB yet) image:

Mat bgrMat = new Mat(mImage.getHeight(), mImage.getWidth(),CvType.CV_8UC4);

I just started learning OpenCV so propably this doesn't have to be 4-channel Mat and instead could be 3-channel but it works for me. Now I use convert color method to change my yuv Mat into bgr Mat.

Imgproc.cvtColor(mYuvMat, bgrMat, Imgproc.COLOR_YUV2BGR_I420);

Now we can do all the image processing like finding contours, colors, circles, etc. To print image back on screen we need to convert it to bitmap:

Mat rgbaMatOut = new Mat();
Imgproc.cvtColor(bgrMat, rgbaMatOut, Imgproc.COLOR_BGR2RGBA, 0);
final Bitmap bitmap = Bitmap.createBitmap(bgrMat.cols(), bgrMat.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(rgbaMatOut, bitmap);

I have all my image processing in seperate thread so to set my ImageView I need to do this on the UI thread.

runOnUiThread(new Runnable() {
                    @Override
                    public void run() {
                        if(bitmap != null) {
                            mImageView.setImageBitmap(bitmap);
                        }
                    }
                });

Bactericide

Answer 3

Have you tried using this script? It's an answer posted by yydcdut on this question

https://github.com/pinguo-yuyidong/Camera2/blob/master/camera2/src/main/rs/yuv2rgb.rs

Torso

Answer 4

Use Shyam Kumar's answer is not right for my phone, but Daniel Więcek's is right.I debug it, find planes[i].getRowStride() is 1216, planes[i].getPixelStride() is 2. While image width and height is both 1200.

Because my reputation is 3, so I cann't comment but post an answer.

Lotetgaronne

You can find the optimized Java convertor from Image to NV21 (which can be fed to OpenCV) here: https://mcmap.net/q/429273/-camera2-captured-picture-conversion-from-yuv_420_888-to-nv21. Note that in most cases, renderscript conversion will be faster than CPU, and not require OpenCV. – Platonism

Answer 5

Approximately 10 times faster than the mentioned "imageToMat"-Function above is this code:

Image image = reader.acquireLatestImage();
...
Mat yuv = new Mat(image.getHeight() + image.getHeight() / 2, image.getWidth(), CvType.CV_8UC1);
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
final byte[] data = new byte[buffer.limit()];
buffer.get(data);
yuv.put(0, 0, data);
...
image.close();

Kiowa

I don't see how your answer can be correct...imageToMat and your simplier function which I've tested produce two different byte arrays. It would be good if OpenCV could handle this new YUV format from Camera2 api. But the next conversion: Imgproc.cvtColor(mYuvMat, bgrMat, mgproc.COLOR_YUV2BGR_I420); is crucial and for all my testing only imageToMat produces good results. If there were YUV2BGR_CAMERA2_API your answer would be better. And maybe now it is. It's two months later now ;) – Zwieback

This might work on some devices, where yBuffer (i.e. image.getPlanes(0)) is actually the full NV21 direct buffer. It is rarely the case. – Platonism

Answer 6

So I ran into the exact same problem, where I had a code that took the old YUV_420_SP format byte[] data from OnPreviewFrame() and converted it to RGB.

They key here is that the 'old' data in the byte[] is as YYYYYY...CrCbCrCbCrCb, and the 'new' data from the Camera2 API is divided into 3 planes: 0=Y, 1=Cb, 2=Cr., from where you can obtain each's byte[]s. So, all you need to do is reorder the new data as a single array that matches the 'old' format, which you can pass to your existing toRGB() functions:

Image.Plane[] planes = image.getPlanes(); // in YUV220_888 format
int acc = 0, i;
ByteBuffer[] buff = new ByteBuffer[planes.length];

for (i = 0; i < planes.length; i++) {
   buff[i] = planes[i].getBuffer();
   acc += buff[i].capacity();
}
byte[] data = new byte[acc],
   tmpCb = new byte[buff[1].capacity()] , tmpCr = new byte[buff[2].capacity()];

buff[0].get(data, 0, buff[0].capacity()); // Y
acc = buff[0].capacity();

buff[2].get(tmpCr, 0, buff[2].capacity()); // Cr
buff[1].get(tmpCb, 0, buff[1].capacity()); // Cb

for (i=0; i<tmpCb.length; i++) {
    data[acc] = tmpCr[i];
    data[acc + 1] = tmpCb[i];
    acc++;
}

..and now data[] is formatted just as the old YUV_420_SP.

(hope that it helps someone, despite the years passed..)

Menado

Your answer looks so promising and indeed get me from crashing to a converted bitmap. Sadly, the generated bitmap is full of green and grey strides only... – Sinter

Note that this only works when the underlying buffers are YV12, i.e. when the uBuffer and vBuffer have pixelStride=1. – Platonism

Answer 7

private Mat ImageToMat(Android.Media.Image image)  
  
{
 int width = 0;
    int height = 0;
    byte[] nv21 = null;

    Android.Media.Image.Plane[] planes = image.GetPlanes();
    width = image.Width;
    height = image.Height;

    ByteBuffer yBuffer = planes[0].Buffer;
    ByteBuffer uBuffer = planes[1].Buffer;
    ByteBuffer vBuffer = planes[2].Buffer;

    int ySize = yBuffer.Remaining();
    int uSize = uBuffer.Remaining();
    int vSize = vBuffer.Remaining();

    nv21 = new byte[ySize + uSize + vSize];

    yBuffer.Get(nv21, 0, ySize);
    vBuffer.Get(nv21, ySize, vSize);
    uBuffer.Get(nv21, ySize + vSize, uSize);

    //yuv.Release();
    Mat yuv = new Mat(height + height / 2, width, CvType.Cv8uc1);
    yuv.Put(0, 0, nv21);

    Mat rgb = new Mat();
    Imgproc.CvtColor(yuv, rgb, Imgproc.ColorYuv2rgbNv21, 3);

    return rgb;
}

Caesarea