Is there a way to crop Image/ImageProxy (before passing to MLKit's analyzer)?

Question

I'm using CameraX's Analyzer use case with the MLKit's BarcodeScanner. I would like to crop portion of the image received from the camera, before passing it to the scanner.

What I'm doing right now is I convert ImageProxy (that I recieve in the Analyzer) to a Bitmap, crop it and then pass it to the BarcodeScanner. The downside is that it's not a very fast and efficient process.

I've also noticed the warning I get in the Logcat when running this code:

ML Kit has detected that you seem to pass camera frames to the detector as a Bitmap object. This is inefficient. Please use YUV_420_888 format for camera2 API or NV21 format for (legacy) camera API and directly pass down the byte array to ML Kit.

It would be nice to not to do ImageProxy conversion, but how do I crop the rectangle I want to analyze?

What I've already tried is to set a cropRect field of the Image (imageProxy.image.cropRect) class, but it doesn't seem to affect the end result.

Posthorse

Although a separate issue, I'm also struggling with modifying things related to this imageProxy. Have you made any progress on this? – Angelita

@Angelita I haven't, unfortunately. At the moment I stick to converting to bitmap and then applying cropping on it, rotating, etc – Posthorse

I'm gaining some ground. Not sure about your particular implementation, but if you are using the default sample code, have you tried using the ImageProxy.setCropRect() method? For example, you could put imageProxy.setCropRect(myCroppingRect) between lines 406 and 407 here. – Angelita

@Angelita I'm not using this sample, but yeah, I've tried the setCropRect method but didn't seem to have any effect. According to a description of the method it sets the "region of valid pixels", but I haven't been able to find what exactly does this mean, as nothing has changed after setting it and barcode detector continued to detect barcodes outside of the specified crop rectangle. – Posthorse

Where are you finding such a description? Are you creating your own custom ImageProxy class? From what I can see via ctrl+clicking up the inheritance tree in the sample code, Android Studio points me to an ImageProxy Interface with an empty setCropRect method with a much less descriptive javadoc. The only docs I can find on ImageProxy are also interfaces without such a description. Where are you finding said imageProxy.image.cropRect class and how are you accessing/creating your imageProxy instance? – Angelita

@Angelita Ah, yeah, sorry, the method description I've mentioned is from Image.setCropRect(). ImageProxy is an interface indeed. From what I can see it has 3 main implementations: AndroidImageProxy, SettableImageProxy, ForwardingImageProxy . First two of these just do mImage.setCropRect(rect) inside the setCropRect() method implementation. And the mImage is an Image, so this was my best bet at finding the meaning of the cropRect. – Posthorse

@Angelita Another thing is, the only place getCropRect() seems to be used, is in ImageUtil.yuvImageToJpegByteArray() and ImageUtil.jpegImageToJpegByteArray(). So it doesn't look like it setting cropRect to ImageProxy does anything on it's own, aside from holding the information about the desired crop rect – Posthorse

I don't see those implementations in the sample code for mlkit. Do you have a minimal working example of the code you are trying to debug? Also looks to be some helpful items here and here – Angelita

@Angelita Thanks, I'll have a look at those SO questions. The interface implementations I've mentioned though are not a part of the sample app, they're a part of androidx.camera.core. MySampleApp – Posthorse

Answer 1

Yes, it's true that if you use ViewPort and set viewport to yours UseCases(imageCapture or imageAnalysis as here https://developer.android.com/training/camerax/configuration) you can get only information about crop rectangle especially if you use ImageAnalysis(because if you use imageCapture, for on-disk the image is cropped before saving and it doesn't work for ImageAnalysis and if you use imageCapture without saving on disk) and here solution how I solved this problem:

1. First of all set view port for use cases as here: https://developer.android.com/training/camerax/configuration

2. Get cropped bitmap to analyze

   override fun analyze(imageProxy: ImageProxy) {
        val mediaImage = imageProxy.image
        if (mediaImage != null && mediaImage.format == ImageFormat.YUV_420_888) {
            croppedBitmap(mediaImage, imageProxy.cropRect).let { bitmap ->
                requestDetectInImage(InputImage.fromBitmap(bitmap, rotation))
                    .addOnCompleteListener { imageProxy.close() }
            }
        } else {
            imageProxy.close()
        }
    }
   
    private fun croppedBitmap(mediaImage: Image, cropRect: Rect): Bitmap {
        val yBuffer = mediaImage.planes[0].buffer // Y
        val vuBuffer = mediaImage.planes[2].buffer // VU
   
        val ySize = yBuffer.remaining()
        val vuSize = vuBuffer.remaining()
   
        val nv21 = ByteArray(ySize + vuSize)
   
        yBuffer.get(nv21, 0, ySize)
        vuBuffer.get(nv21, ySize, vuSize)
   
        val yuvImage = YuvImage(nv21, ImageFormat.NV21, mediaImage.width, mediaImage.height, null)
        val outputStream = ByteArrayOutputStream()
        yuvImage.compressToJpeg(cropRect, 100, outputStream)
        val imageBytes = outputStream.toByteArray()
   
        return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
    }
   

Possibly there is a loss in conversion speed, but on my devices I did not notice the difference. I set 100 quality in method compressToJpeg, but mb if set less quality it can improve speed, it need test.

upd: May 02 '21 :

I found another way without convert to jpeg and then to bitmap. This should be a faster way.

1. Set viewport as previous.

2. Convert YUV_420_888 to NV21, then crop and analyze.

    override fun analyze(imageProxy: ImageProxy) {
        val mediaImage = imageProxy.image
        if (mediaImage != null && mediaImage.format == ImageFormat.YUV_420_888) {
            croppedNV21(mediaImage, imageProxy.cropRect).let { byteArray ->
                requestDetectInImage(
                    InputImage.fromByteArray(
                        byteArray,
                        imageProxy.cropRect.width(),
                        imageProxy.cropRect.height(),
                        rotation,
                        IMAGE_FORMAT_NV21,
                    )
                )
                    .addOnCompleteListener { imageProxy.close() }
            }
        } else {
            imageProxy.close()
        }
    }
   
    private fun croppedNV21(mediaImage: Image, cropRect: Rect): ByteArray {
        val yBuffer = mediaImage.planes[0].buffer // Y
        val vuBuffer = mediaImage.planes[2].buffer // VU
   
        val ySize = yBuffer.remaining()
        val vuSize = vuBuffer.remaining()
   
        val nv21 = ByteArray(ySize + vuSize)
   
        yBuffer.get(nv21, 0, ySize)
        vuBuffer.get(nv21, ySize, vuSize)
   
        return cropByteArray(nv21, mediaImage.width, cropRect)
    }
   
    private fun cropByteArray(array: ByteArray, imageWidth: Int, cropRect: Rect): ByteArray {
        val croppedArray = ByteArray(cropRect.width() * cropRect.height())
        var i = 0
        array.forEachIndexed { index, byte ->
            val x = index % imageWidth
            val y = index / imageWidth
   
            if (cropRect.left <= x && x < cropRect.right && cropRect.top <= y && y < cropRect.bottom) {
                croppedArray[i] = byte
                i++
            }
        }
   
        return croppedArray
    }
   

First crop fun I took from here: Android: How to crop images using CameraX?

And I found also another crop fun, it seems that it is more complicated:

private fun cropByteArray(src: ByteArray, width: Int, height: Int, cropRect: Rect, ): ByteArray {
    val x = cropRect.left * 2 / 2
    val y = cropRect.top * 2 / 2
    val w = cropRect.width() * 2 / 2
    val h = cropRect.height() * 2 / 2
    val yUnit = w * h
    val uv = yUnit / 2
    val nData = ByteArray(yUnit + uv)
    val uvIndexDst = w * h - y / 2 * w
    val uvIndexSrc = width * height + x
    var srcPos0 = y * width
    var destPos0 = 0
    var uvSrcPos0 = uvIndexSrc
    var uvDestPos0 = uvIndexDst
    for (i in y until y + h) {
        System.arraycopy(src, srcPos0 + x, nData, destPos0, w) //y memory block copy
        srcPos0 += width
        destPos0 += w
        if (i and 1 == 0) {
            System.arraycopy(src, uvSrcPos0, nData, uvDestPos0, w) //uv memory block copy
            uvSrcPos0 += width
            uvDestPos0 += w
        }
    }
    return nData
}

Second crop fun I took from here: https://www.programmersought.com/article/75461140907/

I would be glad if someone can help improve the code.

Scenarist

I am not sure how it can work as YUV images have 3 planes, and you are only getting the Y and V planes, not U, i.e. mediaImage.planes[1].buffer is missing here, so it would seem that your resulting image would be missing some colours – Enos

As @AdamBurley mentions this answer doesn't work. This convertion can't be done with only 2 planes, I am getting bizarre colors everywhere. Neither the cropping works. This other answer looks more promising https://mcmap.net/q/429273/-camera2-captured-picture-conversion-from-yuv_420_888-to-nv21 – Doormat

I did look into it further after the comment above. the solution posted does work, but only on certain devices which interleave the U and V planes. on such devices, the U buffer will consist of bytes U1, V1, U2, V2 etc. & the V buffer will be V1, U1, V2, U2, etc. however this is not always the case, some devices don't interleave the buffers at all, in which case the posted code won't work. the proper way to address it is to take into account the pixelStride property of the plane, which tells you the number of pixels between each byte for that plane (i.e. 2 bytes for interleaved, or 1 for non) – Enos

@adam-burley if I add U buffer separately - it will fix problem? – Scenarist

Ok, I will check answer https://mcmap.net/q/429273/-camera2-captured-picture-conversion-from-yuv_420_888-to-nv21 and then fix my answer. – Scenarist

@AlexF. yes, you need to have separate processing for the U buffer. I would not call it "add U buffer separately", because U and V buffers still need to be interleaved in the output, even if they are not interleaved in the input. also need to consider pixel stride for both U and V buffers – Enos

"Convert YUV_420_888 to NV21, then crop and analyze". This didn't work for me I got the image in bizarre colors as @Doormat mentioned above – Krol

Answer 2

I'm still improving the way to do it. But this will work for me now

CameraX crop image before sending to analyze

<androidx.constraintlayout.widget.ConstraintLayout
  android:layout_width="match_parent"
  android:layout_height="wrap_content"
  android:paddingBottom="@dimen/_40sdp">

  <androidx.camera.view.PreviewView
      android:id="@+id/previewView"
      android:layout_width="match_parent"
      android:layout_height="0dp"
      app:layout_constraintDimensionRatio="1:1"
      app:layout_constraintEnd_toEndOf="parent"
      app:layout_constraintStart_toStartOf="parent"
      app:layout_constraintTop_toTopOf="parent" /></androidx.constraintlayout.widget.ConstraintLayout>

Cropping an image into 1:1 before passing it to analyze

                override fun onCaptureSuccess(image: ImageProxy) {
                    super.onCaptureSuccess(image)

                    var bitmap: Bitmap = imageProxyToBitmap(image)
                    val dimension: Int = min(bitmap.width, bitmap.height)
                    bitmap = ThumbnailUtils.extractThumbnail(bitmap, dimension, dimension)

                    imageView.setImageBitmap(bitmap) //Here you can pass the crop[from the center] image to analyze

                    image.close()

                }

**Function for converting into bitmap **

    private fun imageProxyToBitmap(image: ImageProxy): Bitmap {
    val buffer: ByteBuffer = image.planes[0].buffer
    val bytes = ByteArray(buffer.remaining())
    buffer.get(bytes)
    return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}

Spat

Answer 3

You would use ImageProxy.SetCroprect to get the rect and then use CropRect to set it. For example if you had imageProxy, you would do : ImageProxy.setCropRect(Rect) and then you would do ImageProxy.CropRect.

Pamella

This is definitely not true. SetCroped method only sets images metadata and not the actual image. – Mayfly