How to use MediaCodec without MediaExtractor for H264

Question

I need to use MediaCodec without the MediaExtractor and I'm reading the file using a FileInputStream. Currently it is not working, it is showing a greenish scrambled image on the screen.

This is the whole source code:

FileInputStream in = new FileInputStream("/sdcard/sample.ts");

String mimeType = "video/avc";
MediaCodec decoder = MediaCodec.createDecoderByType(mimeType);
MediaFormat format = MediaFormat.createVideoFormat(mimeType, 1920, 1080);

byte[] header_sps = { 0, 0, 0, 1, 103, 100, 0, 40, -84, 52, -59, 1, -32, 17, 31, 120, 11, 80, 16, 16, 31, 0, 0, 3, 3, -23, 0, 0, -22, 96, -108 };
byte[] header_pps = { 0, 0, 0, 1, 104, -18, 60, -128 };
format.setByteBuffer("csd-0", ByteBuffer.wrap(header_sps));
format.setByteBuffer("csd-1", ByteBuffer.wrap(header_pps));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger("durationUs", 63446722);

decoder.configure(format, surface, null, 0);
decoder.start();

ByteBuffer[] inputBuffers = decoder.getInputBuffers();
ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
BufferInfo info = new BufferInfo();
boolean isEOS = false;
long startMs = System.currentTimeMillis();

while (!Thread.interrupted()) {
    if (!isEOS) {
        int inIndex = decoder.dequeueInputBuffer(1000);
        if (inIndex >= 0) {
            byte buffer2[] = new byte[18800 * 8 * 8 * 8];
            ByteBuffer buffer = inputBuffers[inIndex];
            int sampleSize;

            sampleSize = in.read(buffer2, 0, 18800 * 4);

            buffer.clear();
            buffer.put(buffer2, 0, sampleSize);
            buffer.clear();

            if (sampleSize < 0) {
                decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                isEOS = true;
            } else {
                decoder.queueInputBuffer(inIndex, 0, sampleSize, 0, 0);
            }
        }
    }

    int outIndex = decoder.dequeueOutputBuffer(info, 10000);
    switch (outIndex) {
    case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
        Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");
        outputBuffers = decoder.getOutputBuffers();
        break;
    case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
        Log.d("DecodeActivity", "New format " + decoder.getOutputFormat());
        break;
    case MediaCodec.INFO_TRY_AGAIN_LATER:
        Log.d("DecodeActivity", "dequeueOutputBuffer timed out! " + info);
        break;
    default:
        ByteBuffer buffer = outputBuffers[outIndex];
        Log.v("DecodeActivity", "We can't use this buffer but render it due to the API limit, " + buffer);

        while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
            try {
                sleep(10);
            } catch (InterruptedException e) {
                e.printStackTrace();
                break;
            }
        }
        decoder.releaseOutputBuffer(outIndex, true);
        break;
    }

    if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
        Log.d("DecodeActivity", "OutputBuffer BUFFER_FLAG_END_OF_STREAM");
        break;
    }
}

decoder.stop();
decoder.release();

If I use the MediaExtractor, everything works fine. I got the SPS/PPS values by looking at the MediaFormat when using MediaExtractor. If I remove the section below, nothing is shown on the screen.

byte[] header_sps = { 0, 0, 0, 1, 103, 100, 0, 40, -84, 52, -59, 1, -32, 17, 31, 120, 11, 80, 16, 16, 31, 0, 0, 3, 3, -23, 0, 0, -22, 96, -108 };
byte[] header_pps = { 0, 0, 0, 1, 104, -18, 60, -128 };
format.setByteBuffer("csd-0", ByteBuffer.wrap(header_sps));
format.setByteBuffer("csd-1", ByteBuffer.wrap(header_pps));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger("durationUs", 63446722);

What I am missing? How can I get the SPS/PPS values programatically without MediaExtractor?

Garver

could you share with us the corrected code please?? – Postorbital

Are SPS and PPS values you used standard values? In my case, I am getting the data in the form of NAL units, so i am searching for frames with NAL type 7 and 8 for PPS and SPS. Instead can I use the above values directly? Also, I did not understand the posted answer clearly. Can you explain what exactly was the problem with your code? I am using this code as a reference for my scenario. – Whencesoever

No, each stream has its own SPS/PPS values, you should get them like you mentioned above. The problem was that I was sending to MediaCodec (queueInputBuffer) a fixed-size buffer (with size 18800 * 4), but instead you should send complete video frames only. Is it clear now? – Garver

There is a way to exclude MediaExtractor and put byte[] frame (one by one) directly to the inputBuffer (ByteBuffer). Is this what you actually want? – Karb

Answer 1

I'm assuming you're reading a raw H.264 elementary stream and not an MP4 file.

It looks like you're feeding fixed-size blocks of data to the decoder. That doesn't work. You need to put a single access unit into each buffer.

Machination

That was the problem! Now I'm feeding the decoder only with video packets and it is working! – Garver

@Garver Can you post the solution? How you solved this.. I am stuck at decoder.dequeueOutputBuffer(info, 10000); I always get -1... – Nissy

Answer 2

To your last question i.e. how can you get SPS and PPS values, you need to have a mechanism to read the same from the file.

If you are having an elementary stream file, then you would need to parse the file, identify NALU headers and extract the content.

If you have container format, you will need to have a mechanism to read the file format of the container type and extract the information.

If you have a streaming input, then you can receive the content from the incoming SDP information.

As for your current code, I would recommend concatenating both SPS and PPS into one buffer and provide the same to the underlying codec as shown below

byte[] csd_info = { 0, 0, 0, 1, 103, 100, 0, 40, -84, 52, -59, 1, -32, 17, 31, 120, 11, 80, 16, 16, 31, 0, 0, 3, 3, -23, 0, 0, -22, 96, -108, 0, 0, 0, 1, 104, -18, 60, -128 };
format.setByteBuffer("csd-0", ByteBuffer.wrap(csd_info));
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger("durationUs", 63446722);

Erection

The main issue was fixed with the other (accepted) answer. But thanks for answering the second question! +1 for that! – Garver

MediaCodec codecs appear to differ among devices about what constitutes a NAL unit. Some want the 0001 pattern at the beginning, others not. Others are fussy about what you do with non-VCL units. – Illimani

@PaulSteckler.. It's the implementation of the decoders which is making a difference. Some codecs stick to RTP standard and don't handle a start code. An example of this is H.264 streaming to Quicktime which doesn't display the video if a start code is present as RTP standard mandates that first byte is the NALUtype. Ideally, a decoder should be capable of searching for a start code and be robust enough to handle variations. – Erection