Converting YUV_420_888 to JPEG and saving file results distorted image

Question

I've used the ImageUtil class provided in https://mcmap.net/q/502768/-android-camera2-api-yuv_420_888-to-jpeg within my git repo: https://github.com/ahasbini/cameraview/tree/camera_preview_imp (note the implementation is in camera_preview_imp branch) to implement a frame preview callback. An ImageReader is set to preview frames in the ImageFormat.YUV_420_888 format which will be converted into ImageFormat.JPEG using the ImageUtil class and send it to the frame callback. The demo app saves a frame from the callback to a file every 50 frames. All of the saved frame images are coming out distorted similar to below:

If I've changed the ImageReader to use ImageFormat.JPEG instead by doing the following changes in Camera2:

mPreviewImageReader = ImageReader.newInstance(previewSize.getWidth(),
    previewSize.getHeight(), ImageFormat.JPEG, /* maxImages */ 2);
mCamera.createCaptureSession(Arrays.asList(surface, mPreviewImageReader.getSurface()),
    mSessionCallback, null);

the image is coming properly without any distortions however the frame rate drops significantly and the view starts to lag. Hence I believe the ImageUtil class is not converting properly.

Ravel

final image with distortion is image written in a file? – Bodycheck

My bad, edited question to clear confusion. – Ravel

where I can see onImageAvailable(ImageReader reader) (ImageReader.OnImageAvailableListener) method? – Bodycheck

in Camera2 class within the mOnPreviewAvailableListener variable. – Ravel

link please :) I can't find it. – Bodycheck

there you go: github.com/ahasbini/cameraview/blob/camera_preview_imp/library/… – Ravel

Let us continue this discussion in chat. – Ravel

Hi, I have the same problem, How did u fix it?? – Bodice

Answer 1

Solution provided by @volodymyr-kulyk does not take into consideration the row stride of the planes within the image. Below code does the trick (image is of android.media.Image type):

data = NV21toJPEG(YUV420toNV21(image), image.getWidth(), image.getHeight(), 100);

And the implementations:

private static byte[] NV21toJPEG(byte[] nv21, int width, int height, int quality) {
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    YuvImage yuv = new YuvImage(nv21, ImageFormat.NV21, width, height, null);
    yuv.compressToJpeg(new Rect(0, 0, width, height), quality, out);
    return out.toByteArray();
}

private static byte[] YUV420toNV21(Image image) {
    Rect crop = image.getCropRect();
    int format = image.getFormat();
    int width = crop.width();
    int height = crop.height();
    Image.Plane[] planes = image.getPlanes();
    byte[] data = new byte[width * height * ImageFormat.getBitsPerPixel(format) / 8];
    byte[] rowData = new byte[planes[0].getRowStride()];

    int channelOffset = 0;
    int outputStride = 1;
    for (int i = 0; i < planes.length; i++) {
        switch (i) {
            case 0:
                channelOffset = 0;
                outputStride = 1;
                break;
            case 1:
                channelOffset = width * height + 1;
                outputStride = 2;
                break;
            case 2:
                channelOffset = width * height;
                outputStride = 2;
                break;
        }

        ByteBuffer buffer = planes[i].getBuffer();
        int rowStride = planes[i].getRowStride();
        int pixelStride = planes[i].getPixelStride();

        int shift = (i == 0) ? 0 : 1;
        int w = width >> shift;
        int h = height >> shift;
        buffer.position(rowStride * (crop.top >> shift) + pixelStride * (crop.left >> shift));
        for (int row = 0; row < h; row++) {
            int length;
            if (pixelStride == 1 && outputStride == 1) {
                length = w;
                buffer.get(data, channelOffset, length);
                channelOffset += length;
            } else {
                length = (w - 1) * pixelStride + 1;
                buffer.get(rowData, 0, length);
                for (int col = 0; col < w; col++) {
                    data[channelOffset] = rowData[col * pixelStride];
                    channelOffset += outputStride;
                }
            }
            if (row < h - 1) {
                buffer.position(buffer.position() + rowStride - length);
            }
        }
    }
    return data;
}

Method was gotten from the following link.

Ravel

I struggled for a whole afternoon for those strides until I reach this post. I wish to give you 100 upvotes!!!! – Maggot

By the way, the conversion is a bit slow, it takes in my device from 40ms~140ms for each frame. – Maggot

@SiraLam thanks for the feedback. You could open a bounty on the question and award it the answer if you'd like to ;P. With regards to the processing yes ur right it is a slow process, image conversation tends to be like that by nature. To be able to process more frames faster that would require some level of multithreading with some synchronization between the threads to achieve a pipelining effect. Since frame processing is around 100 ms, user wouldn't really feel a lag if viewing the camera stream – Ravel

I would consider that :P Well, user won't feel any lag on the preview since my preview is using another surface different from what I use for these frames. Once I use another thread to do the conversion work, the preview is still very smooth. In fact I am using these frames to do face detections and overlay something on those faces... And sad that using Camera 1 API this is super easy and fast at the same time :( – Maggot

Xiaomi Mi A1, app was crashing using JPEG format for image reader, converted to YUV_420_888 then used your method. Super thanks. – Whitechapel

Great answer! But for some reason, the converted jpeg for me is rotated 90 deg clockwise. Anyone else experienced same thing? – Athelstan

@Athelstan i am facing same issue.Did you find any solution? – Dalhousie

This works perfectly on Xiaomi POCO M3 which otherwise would provide a corrupt NV21 byte array to the face detector. Thank you! – Ilia

I've searched the whole internet, found every possible solution, native, renderscript, you name it. This is the only code which creates proper image on every device (so far) and is also the fastest. – Caricature

Answer 2

Updated ImageUtil:

public final class ImageUtil {

    public static byte[] NV21toJPEG(byte[] nv21, int width, int height, int quality) {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        YuvImage yuv = new YuvImage(nv21, ImageFormat.NV21, width, height, null);
        yuv.compressToJpeg(new Rect(0, 0, width, height), quality, out);
        return out.toByteArray();
    }

    // nv12: true = NV12, false = NV21
    public static byte[] YUV_420_888toNV(ByteBuffer yBuffer, ByteBuffer uBuffer, ByteBuffer vBuffer, boolean nv12) {
        byte[] nv;

        int ySize = yBuffer.remaining();
        int uSize = uBuffer.remaining();
        int vSize = vBuffer.remaining();

        nv = new byte[ySize + uSize + vSize];

        yBuffer.get(nv, 0, ySize);
        if (nv12) {//U and V are swapped
            vBuffer.get(nv, ySize, vSize);
            uBuffer.get(nv, ySize + vSize, uSize);
        } else {
            uBuffer.get(nv, ySize , uSize);
            vBuffer.get(nv, ySize + uSize, vSize);
        }
        return nv;
    }

    public static byte[] YUV_420_888toI420SemiPlanar(ByteBuffer yBuffer, ByteBuffer uBuffer, ByteBuffer vBuffer,
                                                     int width, int height, boolean deInterleaveUV) {
        byte[] data = YUV_420_888toNV(yBuffer, uBuffer, vBuffer, deInterleaveUV);
        int size = width * height;
        if (deInterleaveUV) {
            byte[] buffer = new byte[3 * width * height / 2];

            // De-interleave U and V
            for (int i = 0; i < size / 4; i += 1) {
                buffer[i] = data[size + 2 * i + 1];
                buffer[size / 4 + i] = data[size + 2 * i];
            }
            System.arraycopy(buffer, 0, data, size, size / 2);
        } else {
            for (int i = size; i < data.length; i += 2) {
                byte b1 = data[i];
                data[i] = data[i + 1];
                data[i + 1] = b1;
            }
        }
        return data;
    }
}

Operations to write in file byte[] data as JPEG:

//image.getPlanes()[0].getBuffer(), image.getPlanes()[1].getBuffer()
//image.getPlanes()[2].getBuffer(), image.getWidth(), image.getHeight()
byte[] nv21 = ImageUtil.YUV_420_888toI420SemiPlanar(yBuffer, uBuffer, vBuffer, width, height, false);
byte[] data = ImageUtil.NV21toJPEG(nv21, width, height, 100);
//now write `data` to file

!!! do not forget to close image after processing !!!

image.close();

Bodycheck

updates posted in chat: chat.stackoverflow.com/rooms/144450/… – Ravel

Answer 3

Camera2 YUV_420_888 to Jpeg in Java(Android):

@Override
public void onImageAvailable(ImageReader reader){
    Image image = null;

    try {
        image = reader.acquireLatestImage();
        if (image != null) {

            byte[] nv21;
            ByteBuffer yBuffer = mImage.getPlanes()[0].getBuffer();
            ByteBuffer uBuffer = mImage.getPlanes()[1].getBuffer();
            ByteBuffer vBuffer = mImage.getPlanes()[2].getBuffer();

            int ySize = yBuffer.remaining();
            int uSize = uBuffer.remaining();
            int vSize = vBuffer.remaining();

            nv21 = new byte[ySize + uSize + vSize];

            //U and V are swapped
            yBuffer.get(nv21, 0, ySize);
            vBuffer.get(nv21, ySize, vSize);
            uBuffer.get(nv21, ySize + vSize, uSize);

            String savingFilepath = getYUV2jpg(nv21);



        }
    } catch (Exception e) {
        Log.w(TAG, e.getMessage());
    }finally{
        image.close();// don't forget to close
    }
}

  public String getYUV2jpg(byte[] data) {
    File imageFile = new File("your parent directory", "picture.jpeg");//no i18n
    BufferedOutputStream bos = null;
    try {
        bos = new BufferedOutputStream(new FileOutputStream(imageFile));
        bos.write(data);
        bos.flush();
        bos.close();
    } catch (IOException e) {

        return e.getMessage();
    } finally {
        try {
            if (bos != null) {
                bos.close();
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

    }
    return imageFile.getAbsolutePath();
}

Note: Handle the image rotation issue.

Akimbo

This doesnt work :( the format is still not jpeg after this. – Guilder

Answer 4

I think there is some confusion here with NV and YV formats of YUV. NV (semi-planar) have interleaved U/V. YV (planar) do not. So the conversions being done here are YV12/21 not NV12/21.

Wolgast