System error capturing the output of a MediaProjection virtual display to an ImageReader

Question

I am working on an application that needs to capture the screen to a bitmap to transmit. I am attempting to use the new Android 5.0 android.media.projection APIs to do the screen capture.

The workflow for this API culminates in a call to

mediaProjection.createVirtualDisplay("Test Screen", WIDTH, HEIGHT, DPI,
   DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, surface, null, null);

In my initial attempt at this capture I sourced the surface object from a SurfaceView. This works correctly; the end result is a tiny duplicate of the display being drawn on-screen (resulting in a Droste Effect)

I thought the feature nearly complete, but I then discovered that SurfaceViews are (from a code standpoint) not readable; you cannot get a bitmap from them.

In looking for other solutions I came across this question which has a very similar goal to mine, and in that thread it is suggested to use an ImageReader instead of a SurfaceView to source the Surface that you pass to the createVirtualDisplay API call.

However, when I change my code to use an ImageReader in lieu of a SurfaceView I get runtime logcat errors (no exceptions), and the callback function for the ImageReader never gets called. The createVirtualDisplay call also returns a seemingly valid VirtualDisplay object.

Here is the logcat:

9230-9270/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: createGraphicBuffer failed
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count
9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count

That second line repeats ~100 times before it stops occurring.

Stepping through on the debugger I see that the first error occurs during the createVirtualDisplay call, and all the others happen some point after execution returns to system code.

The only meaningful result for this error relates to an issue in Kitkat, where the API I am trying to consume does not exist. Nonetheless, I tried the fix suggested here (putting android:hardwareAccelerated="false" in the manifest). This did not change the application's behavior.

How can I "set the buffer count" or otherwise work around this error and get the screen as a bitmap?

P.S. My development platform is the Nexus 6.

The full code block, as requested:

MediaProjection mediaProjection = mgr.getMediaProjection(resultCode, data);
ImageReader ir = ImageReader.newInstance(WIDTH, HEIGHT, ImageFormat.JPEG, 5);
VirtualDisplay v = mediaProjection.createVirtualDisplay("Test Screen", WIDTH, HEIGHT, getApplicationContext().getResources().getDisplayMetrics().densityDpi, DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, ir.getSurface(), null, null);

Edit: Regarding the artifact issue, here is the code I am using to get the bitmap out of the image and display it:

 public void onImageAvailable(ImageReader reader) {
        Image image = null;
        ByteArrayOutputStream bos = null;

        try {
            image = reader.acquireLatestImage();
            if (null == image){
                return;
            }
            bos = new ByteArrayOutputStream();
            final Image.Plane[] planes = image.getPlanes();
            final ByteBuffer buffer = (ByteBuffer) planes[0].getBuffer().rewind();
            final Bitmap bitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), Bitmap.Config.ARGB_8888);
            bitmap.copyPixelsFromBuffer(buffer);
            //bitmap.compress(Bitmap.CompressFormat.WEBP, 50, bos);

            runOnUiThread(new Runnable() {
                public void run() {
                    iv.setImageBitmap(bitmap);
                }
            });

Sartorius

I doubt that I'll be able to help on this personally, but it would be useful if you showed the code that you are using (and is giving you the error), instead of the code that you used to use. Beyond that, you might try fadden's answer on this instead, considering that he worked on Android for ~9 years :-). – Marinmarina

Hi I have the same problem, do you get any solution? – Ripp

@Ripp No. Fadden seems to suggest an alternative approach that might work around this issue, but its complexity would require more time than I had planned to spend on this project. – Sartorius

@Sartorius What did Fadden suggest? Is there a link about this? – Ripp

I find an app called 'Screen Recorder', which can capture screen video without root, so there must be some method that I don't know now. – Ripp

@Ripp See CommonsWare's comment above. – Sartorius

Answer 1

I think I can answer this question now, I met the same problem and after I change ImageFormat.JPEG to PixelFormat.RGBA_8888 everything goes well. It seems ImageFormat.JPEG is not supported.

You need to use the following code to get the correct bitmap:

                    int width = img.getWidth();
                    int height = img.getHeight();
                    int pixelStride = planes[0].getPixelStride();
                    int rowStride = planes[0].getRowStride();
                    int rowPadding = rowStride - pixelStride * width;
                    byte[] newData = new byte[width * height * 4];

                    int offset = 0;
                    bitmap = Bitmap.createBitmap(metrics,width, height, Bitmap.Config.ARGB_8888);
                    ByteBuffer buffer = planes[0].getBuffer();
                    for (int i = 0; i < height; ++i) {
                        for (int j = 0; j < width; ++j) {
                            int pixel = 0;
                            pixel |= (buffer.get(offset) & 0xff) << 16;     // R
                            pixel |= (buffer.get(offset + 1) & 0xff) << 8;  // G
                            pixel |= (buffer.get(offset + 2) & 0xff);       // B
                            pixel |= (buffer.get(offset + 3) & 0xff) << 24; // A
                            bitmap.setPixel(j, i, pixel);
                            offset += pixelStride;
                        }
                        offset += rowPadding;
                    }

From this way, the content of bitmap is what you want.

PS: I really want to say, the doc of android is pretty bad. we need to investigate too much detail to use sdk api correctly.

Ripp

Well, that's progress at least. No more weird system errors, but I get a corrupt image out: i.imgur.com/Mu3R61X.png At least it is recognizable. – Sartorius

@Sartorius I am having the same problem. i.imgur.com/rJEjrYZ.jpg (Charlesjean) did you have the same problem or is your screenshot without problems? – Magee

@Magee I don't have much time work on this, and I haven't try saving the image. – Ripp

The format is RGBA_8888, we need to create bitmap of correct format to save this. Do you guys also create bitmap with RGBA_8888? – Ripp

@Ripp I used an ARGB_8888 bitmap since there is no RGBA_8888 bitmap type in Android, which according to this post is the same thing: groups.google.com/forum/#!topic/android-developers/efm1RqUL51A – Sartorius

@Sartorius The buffer from Image.Plane is not with the same format of buffer bitmap needed. Buffer of Image.Plane is from hardware, there is some padding for some reason, we need to make some conversion of this buffer, and then use it to create bitmap. I will post the code when I have time this weekend. – Ripp

That does it. I can capture the screen as a bitmap now, although having the CPU iterate through every pixel is killing my framerate. I may work on parallelizing this code to utilize all the cores of the Nexus 6. – Sartorius

I'm having the same issue using ImageFormat.YUV_420_888, which in theory should be supported (I saw a test in the Android source that used it with ImageReader) – Peeples

newData is not being used:( Should it be used? – Gasperoni

I'd also love an explanation about these bits corrections. – Gasperoni

I only get a black image :( – Haase

Answer 2

A better way to get the Image from ImageReader is just to create right sized bitmap and use the method copyPixelsFromBuffer(). Create ImageReader as follows:

mImageReader = ImageReader.newInstance(mWidth, mHeight, ImageFormat.RGB_565, 2);

Then you can get the image from mImageReader using the code below.

final Image.Plane[] planes = image.getPlanes();
final ByteBuffer buffer = planes[0].getBuffer();
int offset = 0;
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * mWidth;
// create bitmap
bitmap = Bitmap.createBitmap(mWidth+rowPadding/pixelStride, mHeight, Bitmap.Config.RGB_565);
bitmap.copyPixelsFromBuffer(buffer);
image.close();

I have described the process of capturing screen using MediaProjection API along with the mistakes most people made when getting image from ImageReader in a blog post which you can read if interested.

Magee

I get pixelStride ==0, causing ArithmeticException: divide by zero. – Gasperoni