Using getInputImage with MediaCodec for encoding

Question

Background: I do video file demuxing, decode the video track, apply some changes to frames received, decode and mux them again.

The known issue doing this in Android are the number of vendor specify encoder / decoder color formats. Android 4.3 introduced surfaces to get device independent, but I found it hard to work with them as my frame changing routines require a Canvas to write to.

Since Android 5.0, the use of flexible YUV420 color formats is promising. Jointly with getOutputImage for decoding and getInputImage for encoding, Image objects can be used as format retrieved from a decoding MediaCodec. I got decoding working using getOutputImage and could visualize the result after RGB conversion. For encoding a YUV image and queuing it into a MediaCodec (encoder), there seems to be a missing link however:

After dequeuing an input buffer from MediaCodec

int inputBufferId = encoder.dequeueInputBuffer (5000);

I can get access to a proper image returned by

encoder.getInputImage (inputBufferId);

I fill in the image buffers - which is working too, but I do not see a way to queue the input buffer back into the codec for encoding... There is only a

encoder.queueInputBuffer (inputBufferId, position, size, presentationUs, 0);

method available, but nothing that matches an image. The size required for the call can be retrieved using

ByteBuffer  byteBuffer = encoder.getInputBuffer (inputBufferId);

and

byteBuffer.remaining ();

But this seems to screw up the encoder when called in addition to getInputImage().

Another missing piece of documentation or just something I get wrong?

Chitter

How did you fill the image buffers for the image returned by encoder.getInputImage ? I am working with a Bitmap which I used to draw with canvas, and have converted to NV21, but unsure how to separate the panes for pushing to the image buffer. – Cartouche

Answer 1

This is indeed a bit problematic - the most foolproof way probably is to calculate the maximum distance between the start pointer of any plane in the Image to the last byte of any plane, but you need native code to do this (in order to get the actual pointer values for the direct byte buffers).

A second alternative is to use getInputBuffer as you show, but with one caveat. First call getInputBuffer to get the ByteBuffer and call remaining() on it. (Or perhaps capacity() works better?). Only after this, call getInputImage. The detail is that when calling getInputImage, the ByteBuffer returned by getInputBuffer gets invalidated, and vice versa. (The docs says "After calling this method any ByteBuffer or Image object previously returned for the same input index MUST no longer be used." in MediaCodec.getInputBuffer(int).)

Jinni

Thanks for clarification. I moved the getInputBuffer up and stored the remaining() value for the later call of queueInputBuffer(). The input buffer gets indeed invalidated once I call getInputImage(), but queuing sends the image to the encoder now. – Chitter

I get the byte buffers but their size for U and V planes are different to the image received from the camera (they are smaller). It results in a BufferOverflowException when I write image from camera into the encoder. How can you detect what image format the encoder needs and adjust your own buffers accordingly? – Cartouche

This is blasphemy! This is madness! ... This is ANDROID!! ⚔️🛡️ – Interpose

Answer 2

If you have acquired an image from Camera via ImageReader. And you want to input it to the MediaCodec. So this answer could help you.

class to store yuv data

// I define the following class to hold the image data
 class YUV420(
        val width: Int,
        val height: Int,
        val y: ByteBuffer,
        val u: ByteBuffer,
        val v: ByteBuffer,
        val timestampUs: Long
    )

My media format for the encoder

MediaFormat.createVideoFormat(
            MediaFormat.MIMETYPE_VIDEO_AVC,
            previewImageReader.width, previewImageReader.height
        ).apply {
            setInteger(
                MediaFormat.KEY_COLOR_FORMAT,
                MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible
            )
            setInteger(MediaFormat.KEY_BIT_RATE, 100000)
            setInteger(MediaFormat.KEY_FRAME_RATE, 30)
            setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
        }

Copy data from image reader

 private fun ByteBuffer.makeCopy(): ByteBuffer {
        val buffer = if (isDirect) ByteBuffer.allocateDirect(capacity()) else ByteBuffer.allocate(capacity())

        val position = position()
        val limit = limit()
        rewind()

        buffer.put(this)
        buffer.rewind()

        position(position)
        limit(limit)

        buffer.position(position)
        buffer.limit(limit)

        return buffer
    }

val image = imageReader.acquireNextImage()
    
val yuvImage = YUV420(
    width = image.width,
    height = image.height,
    y = image.planes[0].buffer.makeCopy(),
    u = image.planes[1].buffer.makeCopy(),
    v = image.planes[2].buffer.makeCopy(),
    timestampUs = image.timestamp / 1000
 )

input image data to the MediaCodec

val data: YUV420 = ...
val index: Int = ..
val codec: MediaCodec = ...

codec.getInputImage(index)?.let { codecInputImage ->
                    val y = codecInputImage.planes[0].buffer
                    val u = codecInputImage.planes[1].buffer
                    val v = codecInputImage.planes[2].buffer

                    y.put(data.y)
                    u.put(data.u)
                    v.put(data.v)
                }

                val size = data.width * data.height * 3 / 2
                codec.queueInputBuffer(index, 0,size, data.timestampUs, 0)

Biramous

Note that there are 2 hidden assumptions in this code (that the image format is always YUV 4:2:0 and that the image has no padding). Both assumptions will not always hold. To be robust, an implementation needs to ask image.getFormat() as well as image.getCropRect() and be ready to deal with variations. – Falcate