Android MediaCodec Encode and Decode In Asynchronous Mode

Question

I am trying to decode a video from a file and encode it into a different format with MediaCodec in the new Asynchronous Mode supported in API Level 21 and up (Android OS 5.0 Lollipop).

There are many examples for doing this in Synchronous Mode on sites such as Big Flake, Google's Grafika, and dozens of answers on StackOverflow, but none of them support Asynchronous mode.

I do not need to display the video during the process.

I believe that the general procedure is to read the file with a MediaExtractor as the input to a MediaCodec(decoder), allow the output of the Decoder to render into a Surface that is also the shared input into a MediaCodec(encoder), and then finally to write the Encoder output file via a MediaMuxer. The Surface is created during setup of the Encoder and shared with the Decoder.

I can Decode the video into a TextureView, but sharing the Surface with the Encoder instead of the screen has not been successful.

I setup MediaCodec.Callback()s for both of my codecs. I believe that an issues is that I do not know what to do in the Encoder's callback's onInputBufferAvailable() function. I do not what to (or know how to) copy data from the Surface into the Encoder - that should happen automatically (as is done on the Decoder output with codec.releaseOutputBuffer(outputBufferId, true);). Yet, I believe that onInputBufferAvailable requires a call to codec.queueInputBuffer in order to function. I just don't know how to set the parameters without getting data from something like a MediaExtractor as used on the Decode side.

If you have an Example that opens up a video file, decodes it, encodes it to a different resolution or format using the asynchronous MediaCodec callbacks, and then saves it as a file, please share your sample code.

=== EDIT ===

Here is a working example in synchronous mode of what I am trying to do in asynchronous mode: ExtractDecodeEditEncodeMuxTest.java: https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/ExtractDecodeEditEncodeMuxTest.java This example is working in my application

Lanza

Ping, did you see the full example at github.com/mstorsjo/android-decodeencodetest? Is that useful to you, or do you still find it lacking? – Manson

I am impressed @mstorsjo! It appears that you converted the example from which I started to use the non-depreciated async MediaCodec.Callback methods for both encoder and decoder as I had wished. It appears that all of your changes are confined to "ExtractDecodeEditEncodeMuxTest.java". I will report back after I implement. Thank you. – Lanza

Yes, most changes are confined to this file. I had to do some minor changes to InputSurface.java as well though (see github.com/mstorsjo/android-decodeencodetest/commit/…) and one change to OutputSurface.java (github.com/mstorsjo/android-decodeencodetest/commit/…) which no longer was necessary after moving the decoder callbacks to a different thread in github.com/mstorsjo/android-decodeencodetest/commit/…. – Manson

Thank you again @mstorsjo. Do you know why the last frame is not getting processed? This gets tripped every time: assertEquals("encoded and decoded video frame counts should match", mVideoDecodedFrameCount, mVideoEncodedFrameCount); because the encoded count is always one less than the decoded count. – Lanza

Hmm, no, I never saw that on my device, but now when rechecking, I think I have a small race condition at the end, when signaling that the stream has finished. See github.com/mstorsjo/android-decodeencodetest/commit/… for a potential fix. – Manson

Answer 1

I believe you shouldn't need to do anything in the encoder's onInputBufferAvailable() callback - you should not call encoder.queueInputBuffer(). Just as you never call encoder.dequeueInputBuffer() and encoder.queueInputBuffer() manually when doing Surface input encoding in synchronous mode, you shouldn't do it in asynchronous mode either.

When you call decoder.releaseOutputBuffer(outputBufferId, true); (in both synchronous and asynchronous mode), this internally (using the Surface you provided) dequeues an input buffer from the surface, renders the output into it, and enqueues it back to the surface (to the encoder). The only difference between synchronous and asynchronous mode is in how the buffer events are exposed in the public API, but when using Surface input, it uses a different (internal) API to access the same, so synchronous vs asynchronous mode shouldn't matter for this at all.

So as far as I know (although I haven't tried it myself), you should just leave the onInputBufferAvailable() callback empty for the encoder.

EDIT: So, I tried doing this myself, and it's (almost) as simple as described above.

If the encoder input surface is configured directly as output to the decoder (with no SurfaceTexture inbetween), things just work, with a synchronous decode-encode loop converted into an asynchronous one.

If you use SurfaceTexture, however, you may run into a small gotcha. There is an issue with how one waits for frames to arrive to the SurfaceTexture in relation to the calling thread, see https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/DecodeEditEncodeTest.java#106 and https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java#104 and https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/OutputSurface.java#113 for references to this.

The issue, as far as I see it, is in awaitNewImage as in https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/OutputSurface.java#240. If the onFrameAvailable callback is supposed to be called on the main thread, we have an issue if the awaitNewImage call also is run on the main thread. If the onOutputBufferAvailable callbacks also are called on the main thread and you call awaitNewImage from there, we have an issue, since you'll end up waiting for a callback (with a wait() that blocks the whole thread) that can't be run until the current method returns.

So we need to make sure that the onFrameAvailable callbacks come on a different thread than the one that calls awaitNewImage. One pretty simple way of doing this is to create a new separate thread, that does nothing but service the onFrameAvailable callbacks. To do that, you can do e.g. this:

    private HandlerThread mHandlerThread = new HandlerThread("CallbackThread");
    private Handler mHandler;
...
        mHandlerThread.start();
        mHandler = new Handler(mHandlerThread.getLooper());
...
        mSurfaceTexture.setOnFrameAvailableListener(this, mHandler);

I hope this is enough for you to be able to solve your issue, let me know if you need me to edit one of the public examples to implement asynchronous callbacks there.

EDIT2: Also, since the GL rendering might be done from within the onOutputBufferAvailable callback, this might be a different thread than the one that set up the EGL context. So in that case, one needs to release the EGL context in the thread that set it up, like this:

mEGL.eglMakeCurrent(mEGLDisplay, EGL10.EGL_NO_SURFACE, EGL10.EGL_NO_SURFACE, EGL10.EGL_NO_CONTEXT);

And reattach it in the other thread before rendering:

mEGL.eglMakeCurrent(mEGLDisplay, mEGLSurface, mEGLSurface, mEGLContext);

EDIT3: Additionally, if the encoder and decoder callbacks are received on the same thread, the decoder onOutputBufferAvailable that does rendering can block the encoder callbacks from being delivered. If they aren't delivered, the rendering can be blocked infinitely since the encoder don't get the output buffers returned. This can be fixed by making sure the video decoder callbacks are received on a different thread instead, and this avoids the issue with the onFrameAvailable callback instead.

I tried implementing all this on top of ExtractDecodeEditEncodeMuxTest, and got it working seemingly fine, have a look at https://github.com/mstorsjo/android-decodeencodetest. I initially imported the unchanged test, and did the conversion to asynchronous mode and fixes for the tricky details separately, to make it easy to look at the individual fixes in the commit log.

Manson

This is the most comprehensive answer I have seen for encoding, and contains details that have been neglected in many of the monkey-bananna examples out there. The multithreaded GL context switching details are especially advanced - always make sure GL resources are released on the same thread that created them. Neglecting any of these details can lead to ANRs, OOMs, NPEs, Zombies, etc. – Ching

But what kind of improvement can we expect from this approach? I've implemented a decode -> render -> encode pipeline in a single thread (based on Grafika's examples). What I get on a FHD mp4 video is a conversion time that is very close to the video duration (tested on Samsung S7 and S10+, both hardware accelerated). Which is not what I was expecting. But actually even if I try to feed a simple GLES clear to the encoder (without any decoder running) I get similar times. – Scabble

So i started wondering if hardware codecs are just capable tu run slightly faster than playback speed (which will make some kind of sense) or if Grafika's ImputSurface and OutputSurface are somehow blocking the chain maybe due to PST. Can we expect by a good implementation to have a decode-render-encode pipeline running much faster than the playback speed on modern devices? Thankyou – Scabble

Good questions. It depends on resolution of course; if a phone can do 4K decoding/encoding, it probably can only do that barely in realtime. If operating on lower resolutions, it should probably be able to handle much higher speeds, but keep in mind that there's usually quite a bit of latency in the pipeline, so you need to make sure that each stage of the pipeline is saturated all the time, to achieve maximum speed. – Manson

A singlethreaded approach might work just as well, but there's always the tradeoff between too tight polling of all the queues wasting time by ju, vs waiting for too long for an even on one queue when an event happens on another one. If you want to, you could of course try to measure what speed you get with my example (it's been 3 years since I wrote it and I haven't touched it since) vs the original ExtractDecodeEditEncodeMuxTest.java example, if sync vs async matters. – Manson

Thanks for the answer @mstorsjo. Please how would I add presentation times for each of the frames produced by the mediacodec in asynchronous mode. Thanks. – Detoxicate

Presentation timestamps are handled exactly the same in asynchronous mode as in synchronous mode. The linked example above does contain everything necessary to handle it. – Manson

@Manson , thanks so much for responding. So I do not need to manually add it to the output of the async encoder? because when I stream the output to ffmpeg on a pc, ffmpeg does not seem to see any timestamps on the video frames from the h264 output. It reports the timestamps as zero on the video frames and the duration is also N/A – Detoxicate

This is not specific to the async case. If you just send the raw encoder output, there's no timestamps in it. The output from the encoder comes along with a BufferInfo object, that contains the timestamps - the timestamps are then handled by the container (e.g. mp4 file). If you don't use the BufferInfo object's presentationTime field and don't pass the BufferInfo object to e.g. MediaMuxer, then the information is lost. – Manson

Whoops, I suspected so, but I was secretly hoping I was wrong. Thanks a whole lot for clearing this up for me. Could that be why ffmpeg would be reporting the frame rate as 25fps when the video was actually encoded at 30 fps? – Detoxicate

The 25fps default is just a default value that ends up shown when nothing is known. For raw frames without timing info, there's no info about a framerate either - it's just a bunch of frames. – Manson

Answer 2

Can also set the Handler in the MediaEncoder.

---> AudioEncoderCallback(aacSamplePreFrameSize),mHandler);

---

MyAudioCodecWrapper myMediaCodecWrapper;

public MyAudioEncoder(long startRecordWhenNs){
    super.startRecordWhenNs = startRecordWhenNs;
}

@RequiresApi(api = Build.VERSION_CODES.M)
public MyAudioCodecWrapper prepareAudioEncoder(AudioRecord _audioRecord , int aacSamplePreFrameSize)  throws Exception{
    if(_audioRecord==null || aacSamplePreFrameSize<=0)
        throw new Exception();

    audioRecord = _audioRecord;
    Log.d(TAG, "audioRecord:" + audioRecord.getAudioFormat() + ",aacSamplePreFrameSize:" + aacSamplePreFrameSize);

    mHandlerThread.start();
    mHandler = new Handler(mHandlerThread.getLooper());

    MediaFormat audioFormat = new MediaFormat();
    audioFormat.setString(MediaFormat.KEY_MIME, MIMETYPE_AUDIO_AAC);
    //audioFormat.setInteger(MediaFormat.KEY_BIT_RATE, BIT_RATE );
    audioFormat.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
    audioFormat.setInteger(MediaFormat.KEY_SAMPLE_RATE, audioRecord.getSampleRate());//44100
    audioFormat.setInteger(MediaFormat.KEY_CHANNEL_COUNT, audioRecord.getChannelCount());//1(單身道)
    audioFormat.setInteger(MediaFormat.KEY_BIT_RATE, 128000);
    audioFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 16384);
    MediaCodec codec = MediaCodec.createEncoderByType(MIMETYPE_AUDIO_AAC);
    codec.configure(audioFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
    codec.setCallback(new AudioEncoderCallback(aacSamplePreFrameSize),mHandler);
    //codec.start();

    MyAudioCodecWrapper myMediaCodecWrapper = new MyAudioCodecWrapper();
    myMediaCodecWrapper.mediaCodec = codec;

    super.mediaCodec = codec;

    return myMediaCodecWrapper;

}

Body