Reading RGB images with an ImageReader

Question

I'm trying to use an ImageReader to get RGB images from the camera. I'm using the Camera2 API from Android 5.0 "L" on a Nexus 5 running the developer preview. Note that I am not asking how to convert YUV into RGB - I'm trying to get RGB directly from ImageReader. For anyone wanting to know how to convert YUV to RGB, feel free to consult another question, e.g. Convert Android camera2 api YUV_420_888 to RGB.

I already have a SurfaceView configured for RGB images which is working fine, and I believe that the camera hardware produces RGB data (because all the tone mapping and color gain settings on Android are specified to operate on RGB channels).

I can obtain YUV_420_888 images from ImageReader by creating the ImageReader this way:

imageReader = ImageReader.newInstance(W, H, ImageFormat.YUV_420_888, 4);

and then converting the YUV images to RGB. However, this is introducing both an unwanted quantization error (since my application requires RGB images), and unnecessary processing time.

However, when I try creating the image reader this way:

imageReader = ImageReader.newInstance(W, H, PixelFormat.RGB_888, 4);

the image capture fails with the following exception:

java.lang.UnsupportedOperationException: The producer output buffer format 0x22 doesn't match the ImageReader's configured buffer format 0x3.
        at android.media.ImageReader.nativeImageSetup(Native Method)
        at android.media.ImageReader.acquireNextSurfaceImage(ImageReader.java:293)
        at android.media.ImageReader.acquireNextImage(ImageReader.java:339)
        at android.media.ImageReader.acquireLatestImage(ImageReader.java:243)
        at <my code...>

I'm confused on two fronts. First, the output format mentioned, 0x22, is not in either PixelFormat or ImageFormat. It seems to be some sort of undocumented raw mode, but I can't use ImageReader.newInstance(W, H, 0x22, 4) to capture it (I get java.lang.UnsupportedOperationException: Invalid format specified 34). I would love to capture in the raw format but I can't convince ImageFormat to accept it (and the other raw format ImageFormat.RAW_SENSOR is incredibly slow for some reason).

Second, the SurfaceView is already happily consuming RGB_888 images (as far as I can tell), and putting them directly on the screen. So why isn't ImageReader not accepting RGB images properly? What did I do wrong?

Onslaught

The camera output is always YUV. The conversion to RGB is generally performed by the GL driver, which is receiving the data as an external texture. The point of ImageReader is to provide your app with the relatively raw camera data, without doing a conversion, so RGB is not an option. – Shikari

@Shikari actually the camera output is not always YUV, you can change the output format in the camera parameters, but every camera support the YUV color format (NV12), only some devices support RGB. – Pasto

You should use renderscript for this type of conversion. – Descry

Answer 1

The error exception is because of PixelFormat.RGB_888 that isn't the same as ImageFormat.RGB_888.

PixelFormat.RGB_888 is equivalent to ImageFormat.RGB_565 not the one you used ImageFormat.RGB_888.

You have to use ImageFormat.RGB_888 instead of PixelFormat.RGB_888 like this:

imageReader = ImageReader.newInstance(W, H, ImageFormat.RGB_565, 4);

This allows to capture RGB from camera using ImageReader.

Sphery

I don’t think I follow. There is no RGB_888 in ImageFormat. I don’t want RGB_565 as it’s a 16-bit image format which loses a lot of precision (I only lose ~1.5 bits in YUV-RGB conversion, but I’d lose 16 bits in RGB565). – Onslaught