Android CameraX analyse image stream

Question

I want to analyse a stream of image frames and do some computation on it. However, with CameraX, the ImageAnalysis implementation class seems to get called only once - on camera bind.

My question is: How do I run the analysis on a continuous stream of images - conceptually akin to a video stream?

Following is my camera, preview and analysis setup code:

private void setPreview() {
    ListenableFuture<ProcessCameraProvider> instance = ProcessCameraProvider.getInstance(this);

    Activity self = this;
    instance.addListener(() -> {

        try {
            ProcessCameraProvider cameraProvider = instance.get();
            Preview preview = new Preview.Builder().build();


            ImageAnalysis imageAnalysis = new ImageAnalysis.Builder().build();
            imageAnalysis.setAnalyzer(Executors.newFixedThreadPool(1),
                                                    new ImageAnalyser(new CameraLogListener() {
                                                        @Override
                                                        public void log(String msg) {
                                                            Log.e("Camera log", msg);
                                                        }
                                                    }));

            CameraSelector cameraSelector = new CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build();

            cameraProvider.unbindAll();
            Camera camera = cameraProvider.bindToLifecycle(((LifecycleOwner)self), cameraSelector, preview, imageAnalysis);

            preview.setSurfaceProvider(
                    ((PreviewView)findViewById(R.id.cameraTextureView)).createSurfaceProvider(camera.getCameraInfo()));

        } catch (ExecutionException e) {
            e.printStackTrace();
            Log.e("TAG", "Use case binding failed", e);
        } catch (InterruptedException e) {
            e.printStackTrace();
            Log.e("TAG", "Use case binding failed", e);
        }

    }, ContextCompat.getMainExecutor(this));

}

Following is my ImageAnalysis implementation class:

private class ImageAnalyser implements ImageAnalysis.Analyzer {

    CameraLogListener listener;
    public ImageAnalyser(CameraLogListener listener) {
        this.listener = listener;
    }

    @Override
    public void analyze(@NonNull ImageProxy image) {
        ByteBuffer imageBuffer = image.getPlanes()[0].getBuffer();
        StringBuilder sb = new StringBuilder();
        sb.append("format:" + image.getFormat()).append("\n")
            .append(image.getWidth() + " x " + image.getHeight()).append("\n\n");

        for (int i=0; i<image.getPlanes().length; i++) {
            sb.append("pixel stride:").append(image.getPlanes()[i].getPixelStride())
                .append("\nrow stride:").append(image.getPlanes()[i].getRowStride())
                .append("\n");
        }

        listener.log(sb.toString());

    }
}

Ares

are you getting proper fps without any drop after analyze? @Ares – Peba

Answer 1

I found the problem. At the end of the

  public void analyze(@NonNull ImageProxy image) {}

method, you'd need to call image.close(). Quoting from the documentation:

Before returning from analyze(), close the image reference by calling image.close() to avoid blocking the production of further images (causing the preview to stall) and to avoid potentially dropping images. The method must complete analysis or make a copy instead of passing the image reference beyond the analysis method.

Ares

It works! Thanks. Alpha and beta versions are completely diff implementations. – Peba

Answer 2

Close the ImageProxy object, not the image, as stated on the Image analysis Doc

Release the ImageProxy to CameraX by calling ImageProxy.close(). Note that you shouldn't call the wrapped Media.Image's close function (Media.Image.close()).

Astrahan