How to get bitmap (frames) from video using MediaCodec

Question

I'm trying to get all frames from video file using MediaCodec. If I try display video on SurfaceView, everything is ok. But if surface is null, and when I try get Bitmap from byte array, alwaus get null or runtime exception.

This is my code:

private class PlayerThread extends Thread {
    private MediaExtractor extractor;
    private MediaCodec decoder;
    private Surface surface;

    public PlayerThread(Surface surface) {
        this.surface = surface;
    }

    @Override
    public void run() {
        extractor = new MediaExtractor();
        extractor.setDataSource(videoPath);

        for (int i = 0; i < extractor.getTrackCount(); i++) {
            MediaFormat format = extractor.getTrackFormat(i);
            String mime = format.getString(MediaFormat.KEY_MIME);
            if (mime.startsWith("video/")) {
                extractor.selectTrack(i);
                decoder = MediaCodec.createDecoderByType(mime);
                decoder.configure(format, /*surface*/ null, null, 0);
                break;
            }
        }

        if (decoder == null) {
            Log.e("DecodeActivity", "Can't find video info!");
            return;
        }

        decoder.start();

        ByteBuffer[] inputBuffers = decoder.getInputBuffers();
        ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
        BufferInfo info = new BufferInfo();
        boolean isEOS = false;

        while (!Thread.interrupted()) {
            ++numFrames;
            if (!isEOS) {
                int inIndex = decoder.dequeueInputBuffer(10000);
                if (inIndex >= 0) {
                    ByteBuffer buffer = inputBuffers[inIndex];
                    int sampleSize = extractor.readSampleData(buffer, 0);
                    if (sampleSize < 0) {
                        // We shouldn't stop the playback at this point,
                        // just pass the EOS
                        // flag to decoder, we will get it again from the
                        // dequeueOutputBuffer
                        Log.d("DecodeActivity",
                                "InputBuffer BUFFER_FLAG_END_OF_STREAM");
                        decoder.queueInputBuffer(inIndex, 0, 0, 0,
                                MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                        isEOS = true;
                    } else {
                        decoder.queueInputBuffer(inIndex, 0, sampleSize,
                                extractor.getSampleTime(), 0);
                        extractor.advance();
                    }
                }
            }

            int outIndex = decoder.dequeueOutputBuffer(info, 10000);
            switch (outIndex) {
            case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
                Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");
                outputBuffers = decoder.getOutputBuffers();
                break;
            case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
                Log.d("DecodeActivity",
                        "New format " + decoder.getOutputFormat());
                break;
            case MediaCodec.INFO_TRY_AGAIN_LATER:
                Log.d("DecodeActivity", "dequeueOutputBuffer timed out!");
                break;
            default:
                // I tried get Bitmap on few ways
                //1.
                //ByteBuffer buffer = outputBuffers[outIndex];
                //byte[] ba = new byte[buffer.remaining()];
                //buffer.get(ba);
                //final Bitmap bmp = BitmapFactory.decodeByteArray(ba, 0, ba.length);// this return null
                //2. 
                //ByteBuffer buffer = outputBuffers[outIndex];
                //final Bitmap bmp = Bitmap.createBitmap(1920, 1080, Config.ARGB_8888);//using MediaFormat object I know width and height
                //int a = bmp.getByteCount(); //8294400
                //buffer.rewind();
                //int b = buffer.capacity();//3137536
                //int c = buffer.remaining();//3137536
                //bmp.copyPixelsFromBuffer(buffer); // java.lang.RuntimeException: Buffer not large enough for pixels
                //I know what exception mean, but i don't know why xception occurs

                //In this place I need bitmap

                // We use a very simple clock to keep the video FPS, or the
                // video
                // playback will be too fast
                while (info.presentationTimeUs / 1000 > System
                        .currentTimeMillis() - startMs) {
                    try {
                        sleep(10);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                        break;
                    }
                }
                decoder.releaseOutputBuffer(outIndex, true);
                break;
            }

            // All decoded frames have been rendered, we can stop playing
            // now
            if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                Log.d("DecodeActivity",
                        "OutputBuffer BUFFER_FLAG_END_OF_STREAM");
                break;
            }
        }

        decoder.stop();
        decoder.release();
        extractor.release();
    }
}

I have no idea how to solve it.

Samos

Reformatory

Answer 1

There are a couple of problems with your code (or, arguably, with MediaCodec).

First, the ByteBuffer handling in MediaCodec is poor, so you have to manually set the buffer parameters from the values in the BufferInfo object that is filled out by dequeueOutputBuffer().

Second, the values that come out of the MediaCodec are in YUV format, not RGB. As of Android 4.4, the Android framework does not provide a function that will convert the output to Bitmap. You will need to provide your own YUV-to-RGB converters (plural -- there are multiple formats). Some devices use proprietary, undocumented color formats.

You can see an example of extracting and examining MediaCodec decoder buffer contents in the CTS EncodeDecodeTest buffer-to-buffer methods (e.g. checkFrame()).

A more reliable way to go about this is to go back to decoding to a Surface, but extract the pixels with OpenGL ES. The bigflake ExtractMpegFramesTest shows how to do this.

Harwell

Maybe you can help me https://mcmap.net/q/1164660/-how-to-get-frame-by-frame-from-mp4-mediacodec/5709159 I thinks it is exact question for you... – Deerstalker

Answer 2

The short answer is:

In the default section of your switch statement, you need to reset the ByteBuffer position, so instead of:

ByteBuffer buffer = outputBuffers[outIndex];
byte[] ba = new byte[buffer.remaining()];
buffer.get(ba); 

you should have something like

ByteBuffer buffer = outputBuffers[outIndex];
buffer.position(info.offset);
buffer.limit(info.offset + info.size);
byte[] ba = new byte[buffer.remaining()];
buffer.get(ba); 

In your original code, you will find that your ByteBuffer has a remaining() of 0.

Theodor

Answer 3

It's not trivial, but possible. See this this class: https://github.com/dburckh/HeifViewer/blob/develop/app/src/main/java/com/homesoft/iso/heif/ImageDecoder.kt

The gist of it is you create an ImageReader Surface as the target of the MediaCodec decoder. You then use PixelCopy to copy the Surface to a Bitmap. The problem is if MediaCodec decodes frames faster than you can convert them, you'll drop frames. I resolved this by only feeding MediaCodec a new frame after I had decoded the prior frame to a Bitmap.

Another possible solution is use MediaCodec.getOutputImage(). Assuming the output is YUV, you could then use libyuv to convert to ARGB. All this would be done in C and it assumes you can access the ByteBuffer's in the Image. That method also has warts because there are many different Image formats. FYI: Most video is a YUV420 variant, so that limits things a little.

The easiest way would be to get the HardwareBuffer from the Image and wrap it in a Bitmap. Unfortunately, that is specifically not supported, but it might be worth trying. :)

Pursuant