ML Kit - Android - Text Recognition - Text Orientation

Question

I started a new application for Text Recognition of file numbers onto shelves and I used the following ML Kit Text Recognition on Youtube: Tutorial ML KIT Text Recognition

It works perfectly as long as the file numbers are written horizontally. When the text is oriented vertically and I turn the phone in landscape position, the application take into account the orientation and the textrecognition doesn't work anymore.

I tried to add both

<activity android:name=".NameOfTheActivity" android:screenOrientation="portrait"> 

in the manifest and setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT) or even

setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_NOSENSOR)

into the onCreate method. The activity stays in portrait mode even if the phone is in landscape position. However, the text recognition still doesn't work when using my phone in landscape position.

The corresponding parts of the Activity code are the following :

//Event Camera View
    cameraView.addCameraKitListener(new CameraKitEventListener() {
        @Override
        public void onEvent(CameraKitEvent cameraKitEvent) {
        }

        @Override
        public void onError(CameraKitError cameraKitError) {
        }

        @Override
        public void onImage(CameraKitImage cameraKitImage) {
            //Show Dialog
            waitingDialog.show();

            //Processing image
            Bitmap bitmap = cameraKitImage.getBitmap();
            bitmap = Bitmap.createScaledBitmap(bitmap, cameraView.getWidth(), cameraView.getHeight(), false);
            cameraView.stop();

            recognizeText(bitmap);
        }

        @Override
        public void onVideo(CameraKitVideo cameraKitVideo) {
        }
    });

private void recognizeText(Bitmap bitmap) {
    FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);

    FirebaseVisionTextRecognizer textRecognizer = FirebaseVision.getInstance().getOnDeviceTextRecognizer();

    textRecognizer.processImage(image)
            .addOnSuccessListener(new OnSuccessListener<FirebaseVisionText>() {
                @Override
                public void onSuccess(FirebaseVisionText firebaseVisionText) {
                    drawTextResult(firebaseVisionText);
                }
            }).addOnFailureListener(new OnFailureListener() {
        @Override
        public void onFailure(@NonNull Exception e) {
            Log.d("EDMT_ERROR", e.getMessage());
        }
    });
}

Does anyone has some tip to make it work or knows a tutorial that works ? I have seen an example here : Firebase example in which they use a media.Image object to take into account the orientation. As I don't know how to modify the original code (which use a bitmap ), I was just trying to "deactivate" the orientation sensor so that the image/bitmap taken are exactly the same whether I am using my phone in portrait or landscape position. In that case, the recognizeText(bitmap) method should return the same answers in both case.

Societal

Answer 1

I finally use a bitmap rotate method

    public static Bitmap rotateImage(Bitmap source, float angle) {
    Matrix matrix = new Matrix();
    matrix.postRotate(angle);
    return Bitmap.createBitmap(source, 0, 0, source.getWidth(), source.getHeight(),
            matrix, true);
}

with the following amendments to the original code

FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(rotateImage(bitmap, 270));

and

bitmap = Bitmap.createScaledBitmap(bitmap, cameraView.getHeight(), cameraView.getWidth(), false);

That works... the showed image is still distorded but the analyzed image is OK and return good results in Landscape position of the phone (don't try in portrait position, it won't work anymore).

Societal

Where did you add this in the code? in TextRecognitionProcessor? for working in both landscape and portrait maybe you can use the orientation sensor callback and a condition to rotate the image or no. – Corkhill