To use HW decoder in android through libstagefright, what to set for kKeyAVCC in meta data for frame base decoding instead of MP4 playback?

Question

I have been trying to use libstagefright to decode h264 compressed frames. I don't have MP4 file, instead I want to decode frame by frame. I have been exploring the sample from a link. This sample uses ffmpeg to parse mp4 file. and using ffmpeg's avcodeccontext it set and find the required metadata. Now I want to set kKeyAVCC, kKeyWidth, kKeyHeight, kKeyIsSyncFrame and kKeyTime. I am not clear about each of these parameter. So, all I want to know is whether all these parameter needs to be set? What are the purpose of these parameters and what to set into them for meta data for frame by frame decoding. When I do not set kKeyTime, omxcodec crashes on read mediabuffer. And If I get success on read operation, I am not getting the meta data values I have set in MediaBuffer's derived read method. I get video dimesiones of the frame and error code of INFO_FORMAT_CHANGED.

Folie

Answer 1

When a new codec is created, the metadata is passed from the parser to the decoder as part of the OMXCodec::Create method. I presume in your implementation you would have taken care to pass the metadata in MetaData format as specified in the plain vanilla android implementation.

For example, please refer to AwesomePlayer::initVideoDecoder in which mVideoTrack->getFormat() is invoked to get the metadata of the video track. Please note that this is not part of a MediaBuffer, but is passed as a separate object.

Once the decoder is created, configureCodec is invoked. In this method, OMXCodec reads different configuration parameters to initialize the decoder.

kKeyAVCC corresponds to the Codec Specific Data or csd which is essentially the SPS and PPS of the underlying H.264 stream.

kKeyWidth and kKeyHeight corresponds to the width and height of the video frame. For initializing the decoder, you can set some more additional parameters. For example, if you to set a specific colorFormat for the output of decoder, you can set the same through kKeyColorFormat.

Once the decoder is created, you will have to pass the individual frames through the standard openmax interfaces. The decoder is started with the invocation of OMXCodec::read method which will flood fill the input and output buffers.

The input buffer is filled through the OMXCodec::drainInputBuffer method which reads a MediaBuffer from the parser module (which in your case is your specific module). The content of the MediaBuffer is copied onto the buffer populated on the input port of the OMX component. Along with this data, the timestamp of this buffer is also passed. The timestamp information is read through the kKeyTime parameter which is passed along with the MediaBuffer.

Hence, for every frame which is passed in a MediaBuffer, you need to ensure that a valid timestamp is also passed to the underlying decoder which gets reflected on the output port of the decoder.

In your question, you had queried about kKeyIsSyncFrame. This flag is set by an Encoder as part of the FillBufferDone callback i.e. when an encoder encodes a key frame like IDR frame, then it communicates this information through this specific flag as part of the callback on the output port of the encoder. For decoding, this is not relevant.

If you can post some further logs with OMXCodec logs enabled, it might be easier to provide a more accurate answer.

P.S. In android framework, there is a command line utility called Stagefright which creates a parser and decoder and performs a command line decoding without any rendering. This could be a good reference for you to plugin your own parser.

Throughway

First of all Thank you very much for well detailed answer. You have provided very good information. I am linking libstagfright.so through android.mk in my project. I need to include all corresponding include headers from Android source. I have written two classes called, CustomMediaSource and CustomDecoder. CustomMediaSource is derived from MediaSource and implements read method whereas in CustomDecoder, I am calling OMXCodec::Create method. Now, my doubt is whether libstagefright is stable to use for all android devices. I mean can it very according to vendors like TI, Qualcom etc.? – Folie

From an API perspective, the libstagefright library will be the same irrespective of the vendors. However, if you are looking at some performance numbers, then there could be potential customizations specific to the vendors SoC. I am not saying that this will be the case always, but is probable. If you don't have access to sources, then you may not be able to distinguish. – Throughway

Well Thanks. I want to develop a platform independent android app. So it becomes important to prefer stable apis. So, as you said, I will find libstagefright.so in all android phones and it will have same api structures and I can derive Media Source and use the derived class's instance for OMXCodec::Create method . . . Right? – Folie

Thanks Ganesh. I need your one more help for implementation. You mentioned in your answer that I can set the output colorspace for codec using kKeyColorFormat flag. Now , when I set kKeyColorFormat to OMX_COLOR_Format16bitRGB565, OMXCodec::Create method does not return. As per my requirement I want decode and display h264 frame and surface can render in rgb565 colorspace. So, can you tell me how can I achieve yuv rendering over surface? and what can be wrong when I set kKeyColorFormat parameter to rgb565? – Folie

It is very rare for a Video Decoder to support OMX_Color_Format16BitRGB565 as the output format. To know exactly what is happening, you may require either OMXCodec logs or enable logs from your OMX component. If you wish to render the surface, you will have to modify inside the surfaceflinger and/or hwcomposer interface to perform the colorConversion from YUV to RGB – Throughway

When I was surfing on net for surface rendering from native, I came across this link. You posted it in 2010. And I am finding the same approach for buffer feeding from native all over the internet. But I am facing an issue. When I call GetFieldID(surfaceClass, "mSurface", "I"), error is thrown, says java.lang.NoSuchFieldError: no field with name='mSurface' signature="I" in class Landroid/view/Surface;. Any comment on this? I checked the doc of android.view.surface and there is no mSurface field. So where can I find that(for sp<Surface>)? – Folie

I found the solution for the above issue. Go through this link. It says, find "mNativeSurface" instead of "mSurface" for GetFieldID. It seems like, till Froyo the parameter was mSurface and from Gingerbread onwards, it is mNativeSurface. – Folie

Hello Ganesh, About the colorspace matter, you said its very rare for video decoder to support rgb565. Now, can you please explain me how mp4 files in android is played. Generally mp4 files consist of h264 encoded video frames and aac encoded audio chuks. Then, How does it decode and display h264 frames. Especially, how android's default player manager colorspace conversion for video rendering? Lots of android devices claims the support of 720p and 1080p resolutions. How do they manage colorspace and image scaling and etc rendering stuff for performance? – Folie

This is a very interesting question. For higher resolutions, you would require dedicated hardware support to perform the decoding. If we are considering 720p or 1080p, it is very clear that your SoC should support dedicated hardware blocks. In such advanced SoC, one would either have another post-processor or video processor or some such block, which can perform color conversion and other standard image processing tasks. – Throughway

In short, a vendor would integrate their hardware accelerators to speed up color conversion. The question now comes, where is the color conversion employed. In AwesomePlayer, you could either create a AwesomeNativeWindowRenderer which is the standard one or AwesomeLocalRenderer which is employed for software based codecs. Internally AwesomeLocalRenderer employs SoftwareRenderer. For every render call, there is a call to mConverter->convert which will convert from YUV to RGB. For HW accelerated solutions, this is handled a bit differently, but conceptually remains the same – Throughway

Nicely Explained !!! Thanks. I found that the device on which I am testing does not have any hardware codecs. All the components are omx.google.*.*. So from awesomeplayer, As you mentioned, I explored AwesomeLocalRenderer. And Then I have created an instance of ColorConvertion Class and initiated for yuv420planar to rgb565. Which gave me 10FPS Average for 1080p for medium quality h264 compression and above 10 GOV. P frames takes avg 90ms and I frame takes upto 170ms for decode + display. I still wanted to test for HW decoder. Is there any way to scale the video, say to make it fit to screen? – Folie

@Folie For scaling the video, you would have to setup the SurfaceFlinger / HwComposer by setting the transform properties for the appropriate scaling. – Throughway

@Folie I imitate AwesomePlayer use libstagefright hardware decode video, it's ok on Android 4.0, but on Android 4.1+, it's black screen, Have you meet this problem? – Garrek

I worked with ICS build only. I didn't tested for 4.1+. stagefright apis are variable for version to version. Jelly beans onward, media service provides support to access hardware decoder from Java layer. May be it cause some changes in api. Have you tested different manufacturer's jelly bean ported devices? – Folie

@sam18.. Yes I have tested different JB devices – Throughway