Getting QualComm encoders to work via MediaCodec API

Question

I am trying to do hardware encoding (avc) of NV12 stream using Android MediaCodec API.

When using OMX.qcom.video.encoder.avc, resolutions 1280x720 and 640x480 work fine, while the others (i.e. 640x360, 320x240, 800x480) produce output where chroma component seems shifted (please see snapshot).

I have double-checked that the input image is correct by saving it to a jpeg file. This problem only occurs on QualComm devices (i.e. Samsung Galaxy S4).

Anyone has this working properly? Any additional setup / quirks necessary?

Counterattraction

Android 4.3 added some useful new features, and also added CTS tests that feed YUV data into MediaCodec. The buffer-to-buffer and buffer-to-surface tests in bigflake.com/mediacodec/#EncodeDecodeTest may be informative. – Unmeet

BTW, there's discussion in android-platform Google Group which mentions QualComm encoders: groups.google.com/d/msg/android-platform/awaNwgb6EbY/… One of the commenters states that chroma plane should be aligned by 2048 bytes boundary, but that only works partially for me. Some resolutions are still buggy, 176x144, for example. – Pridgen

Answer 1

Decoder(MediaCodec) has its MediaFormat, it can be received using getOutputFormat. Returned instance can be printed to log. And there you can see some useful information. For example in your case value like "slice-height" could be useful. I suspect that it is equal to height for 1280x720 and 640x480 and differs for other resolutions. Probably you should use this value to get chroma offset.

Mannerism

As a quick fix, I changed the resolutions according to what I saw in 'slice-height' and 'stride' of the decoder. For example, 640x384 instead of 640x360 and 384x256 instead of 320x200. Then it worked fine :) Thanks! – Counterattraction

@badbadboy did you find a way to dynamically calculate stride and sliceHeight given a fixed resolution for the qualcomm encoder? – Worrell

I am wondering when getOutputFormat() should be called? I am getting IllegalStateException when calling it in various places - before starting MediaCodec and after starting it. Documentation states that it should be called after INFO_OUTPUT_FORMAT_CHANGED event, but encoder never sending this one. – Pridgen

Answer 2

Yep, the OMX.qcom.video.encoder.avc does that but not on all devices/android version. On my Nexus 4 with Android 4.3 the encoder works fine, but not on my S3 (running 4.1)

The solution for an S3 running 4.1 with the OMX.qcom.video.encoder.avc (it seems that some S3 have another encoder) is to add 1024 bytes just before the Chroma pane.

// The encoder may need some padding before the Chroma pane
int padding = 1024;                     
if ((mWidth==640 && mHeight==480) || mWidth==1280 && mHeight==720) padding = 0;

// Interleave the U and V channel
System.arraycopy(buffer, 0, tmp, 0, mYSize); // Y
for (i = 0; i < mUVSize; i++) {
   tmp[mYSize + i*2 + padding] = buffer[mYSize + i + mUVSize]; // Cb (U)
   tmp[mYSize + i*2+1 + padding] = buffer[mYSize + i]; // Cr (V)
}
return tmp;

The camera is using YV12 and the encoder COLOR_FormatYUV420SemiPlanar.

Your snapshot shows the same kind of artefacts I had, you may need a similar hack for some resolutions, maybe with another padding length

You should also avoid resolutions that are not a multiple of 16, even on 4.3 apparently (http://code.google.com/p/android/issues/detail?id=37769) !

Becki

how do I determine mYSize and mUVSize ? – Brabazon