Mobile Vision API - concatenate new detector object to continue frame processing

Question

I want to use the new face detection feature that the vision API provides along with additional frame processing in an application. For this, I need to have access to the camera frame that was processed by the face detector, and concatenate a processor using the face detected data.

As I see in the sample, the CameraSource abstracts the detection and camera access, and I can't have access to the frame being processed. Are there examples of how to get the camera frame in this API, or, maybe, create and concatenate a detector that receives it? Is that possible at least?

Thanks, Lucio

Titan

Answer 1

Yes, it is possible. You'd need to create your own subclass of Detector which wraps FaceDetector and executes your extra frame processing code in the detect method. It would look something like this:

class MyFaceDetector extends Detector<Face> {
  private Detector<Face> mDelegate;

  MyFaceDetector(Detector<Face> delegate) {
    mDelegate = delegate;
  }

  public SparseArray<Face> detect(Frame frame) {
    // *** add your custom frame processing code here
    return mDelegate.detect(frame);
  }

  public boolean isOperational() {
    return mDelegate.isOperational();
  }

  public boolean setFocus(int id) {
    return mDelegate.setFocus(id);
  }
}

You'd wrap the face detector with your class, and pass your class into the camera source. It would look something like this:

    FaceDetector faceDetector = new FaceDetector.Builder(context)
            .build();
    MyFaceDetector myFaceDetector = new MyFaceDetector(faceDetector);

    myFaceDetector.setProcessor(/* include your processor here */);

    mCameraSource = new CameraSource.Builder(context, myFaceDetector)
            .build();

Your detector will be called first with the raw frame data.

Note that the image may not be upright, if the device is rotated. You can get the orientation through the frame's metadata.getRotation method.

One word of caution: once the detect method returns, you should not access the frame pixel data. Since the camera source recycles image buffers, the contents of the frame object will be eventually overridden once the method returns.

EDIT: (additional notes) You could also avoid the boilerplate code of MyFaceDetector using a MultiDetector like this:

MultiDetector multiDetector = new MultiDetector.Builder()
    .add(new FaceDetector.Builder(context)
                .build())
    .add(new YourReallyOwnDetector())
    .build();

Also note the use of FaceTrackerFactory in conjuction with MultiProcessor described there.

Turku

No idea how this can still allow you to crop the image. The frame comes in and you can edit it but Frame is private and there is no setBitMap... – Porterhouse

@Porterhouse you will probably have to create a new Frame using Frame.Builder() and set the cropped bitmap into it using setBitmap(Bitmap image) or setImageData(ByteBuffer data, int width, int height, int format). All the params can be obtained directly from the input frame or the frame metadata. – Supplication

Answer 2

Here's the final solution I settled on. It assumes the box is centered on the screen.

public class BoxDetector extends Detector {
    private Detector mDelegate;
    private int mBoxWidth, mBoxHeight;

    public BoxDetector(Detector delegate, int boxWidth, int boxHeight) {
        mDelegate = delegate;
        mBoxWidth = boxWidth;
        mBoxHeight = boxHeight;
    }

    public SparseArray detect(Frame frame) {
        int width = frame.getMetadata().getWidth();
        int height = frame.getMetadata().getHeight();
        int right = (width / 2) + (mBoxHeight / 2);
        int left = (width / 2) - (mBoxHeight / 2);
        int bottom = (height / 2) + (mBoxWidth / 2);
        int top = (height / 2) - (mBoxWidth / 2);

        YuvImage yuvImage = new YuvImage(frame.getGrayscaleImageData().array(), ImageFormat.NV21, width, height, null);
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        yuvImage.compressToJpeg(new Rect(left, top, right, bottom), 100, byteArrayOutputStream);
        byte[] jpegArray = byteArrayOutputStream.toByteArray();
        Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);

        Frame croppedFrame =
                new Frame.Builder()
                        .setBitmap(bitmap)
                        .setRotation(frame.getMetadata().getRotation())
                        .build();

        return mDelegate.detect(croppedFrame);
    }

    public boolean isOperational() {
        return mDelegate.isOperational();
    }

    public boolean setFocus(int id) {
        return mDelegate.setFocus(id);
    }
}

Wrap this class in your detector like this

BarcodeDetector barcodeDetector = new BarcodeDetector.Builder(context).build();
BoxDetector boxDetector = new BoxDetector(barcodeDetector, heightPx, widthPx);

Preciosa

@saturov how should I use detect function? What frame should I pass? I've created BoxDetector: new BoxDetector(barcodeDetector, cameraView.getHeight()/3, cameraView.getWidth()); and when I'm calling boxDetector.detect I don' know what frame to pass. – Talebearer

@Preciosa how use boxDetector ? – Sussex

In case someone else runs into this, I was getting IllegalArgumentException: rectangle is not inside the image The solution was override fun receiveFrame(frame: Frame?) = delegate.receiveFrame(frame) – Pisces

how to detect only face box using Sparsearray? – Chiffonier

Thank you, I was searching solution for the same, and it worked for me. Although I tried for creating bitmap using the OverLayView and CameraSourcePreiview but nothing worked. – Tumbler

I am converting this code to Kotlin. Please share all the imports. – Agace

any exact solution guys? – Radiance

Answer 3

As per user(New Developer) request, how to set box detector. You can use like this

use @MCR BoxDetector class and then follow these steps.

I am just giving example about text Recognizer so you can set like this

TextRecognizer mTextRecognizer = new TextRecognizer.Builder(getApplicationContext()).build();
BoxDetector boxDetector = new BoxDetector(mTextRecognizer, heightPx, widthPx);

set boxDetecotr here

boxDetector.setProcessor(new Detector.Processor<TextBlock>() {
                @Override
                public void release() {

                }

                @Override
                public void receiveDetections(Detector.Detections<TextBlock> detections) {
                    SparseArray<TextBlock> items = detections.getDetectedItems();
                    StringBuilder stringBuilder = new StringBuilder();
                    for (int i = 0; i < items.size(); ++i) {
                        TextBlock item = items.valueAt(i);
                        if (item != null && item.getValue() != null) {
                            stringBuilder.append(item.getValue() + " ");
                        }
                    }

                    final String fullText = stringBuilder.toString();
                    Handler handler = new Handler(Looper.getMainLooper());
                    handler.post(new Runnable() {
                        public void run() {
                            // here full string(fullText) you can get whatever is it scanned.
                        }
                    });

                }
            });

Droop

Hello, I am facing the Exception Detector processor must first be set with setProcessor in order to receive detection results. – Labradorite