Get text content of the android screen

Question

Is there any way to get the text content that is being displayed on the android mobile screen. It may be by any android app.

for example in the image linked above, i wish to get all the text (starting from the "Larry Page" till the end "Fred Smith" and also the position of all the words on the screen) that is in the image and its respective position on the screen.

Canossa

are those texts in your app? – Olen

@phuc_tran seems like he wants to scratch the text from any android app. – Mitra

No, the text won't be from my app, i want to acquire any and every text that is being displayed by any app on the mobile screen. – Canossa

Have you found a solution for this yet? – Airwaves

How this app is doing, though! play.google.com/store/apps/… – Hostler

Answer 1

This can be achieved using the Accessibility feature of Android (as the Universal copy app does).

What you need to do is implement a class called AccessabilityService which can access all the screen UI through a listener.

In order to that you need to declare it in the manifest:

 <service android:name=".MyAccessibilityService">
 <intent-filter>
     <action android:name="android.accessibilityservice.AccessibilityService" />
 </intent-filter>
 <meta-data android:name="android.accessibilityservice" android:resource="@xml/accessibilityservice" />

the file @xml/accessibilityservice is where you configure the service like to which package to listen or what kind of events you want to get:

<accessibility-service
 android:accessibilityEventTypes="typeViewClicked|typeViewFocused"
 android:packageNames="com.example.android.myFirstApp, com.example.android.mySecondApp"
 android:accessibilityFeedbackType="feedbackSpoken"
 android:notificationTimeout="100"
 android:settingsActivity="com.example.android.apis.accessibility.TestBackActivity"
 android:canRetrieveWindowContent="true"/>

In the class override the method:

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
    Log.v(TAG, String.format(
            "onAccessibilityEvent: type = [ %s ], class = [ %s ], package = [ %s ], time = [ %s ], text = [ %s ]",
            getEventType(event), event.getClassName(), event.getPackageName(),
            event.getEventTime(), getEventText(event)));
}

In that method, you should find in the UI the desired text/TextView/EditText and extract from it the text you want.

NOTE: this way required the Accessibility permission which only the user can give you and it cannot be done in runtime.

You can read more in the Android documentation about how to implement an AccessabilityService

EDIT: after the question in the comment I'm adding the code to read from all the text components in the screen:

    private void printAllViews(AccessibilityNodeInfo mNodeInfo) {
        if (mNodeInfo == null) return;
        String log ="";
        for (int i = 0; i < mDebugDepth; i++) {
            log += ".";
        }
        log+="("+mNodeInfo.getText() +" <-- "+ 
        mNodeInfo.getViewIdResourceName()+")";
        Log.d(TAG, log);
        if (mNodeInfo.getChildCount() < 1) return;
        mDebugDepth++;

        for (int i = 0; i < mNodeInfo.getChildCount(); i++) {
            printAllViews(mNodeInfo.getChild(i));
        }
        mDebugDepth--;
    }

Just don't forget to initialize the AccessibilityNodeInfo and the depth counter like so:

public void onAccessibilityEvent(AccessibilityEvent event) {
    mDebugDepth = 0; 
    mNodeInfo = event.getSource();
    printAllViews(mNodeInfo);
 ...
}

Caldarium

Hello. I tried to log like you did but cannot get any text on screen. Can you share more about your work? Thanks. – Jacinto

see the EDIT part. – Caldarium

Do i need to manually create the accessibility services xml? In which folder? – Trust

yes, you need to create that file manually. the path is: /app/src/main/res/xml/accessibilityservice.xml – Caldarium

Why do you use mDebugDepth? Anything more to take care instead of only for logging? Because sometimes i get java.lang.StackOverflowError: stack size 8MB after 10 calls printAllViews() ~ mDebugDepth == 10 ?! – Jacinto

mDebugDepth here means the maximum tree depth you want to go to fetch the nodeInfo. As accessbilityNodeInfo is just a node of the tree(windowContent), while doing a DFS/BFS you need to be sure to not go beyond a recursion depth otherwise your stack will overflow – Stickybeak

Answer 2

This can be done by using "uiautomatorviewer". This bin file can be found inside Android SDK/tools/bin/ folder.

Run the uiautomatorviewer in the terminal.

$./uiautomatorviewer

• Make sure USB debugging is enabled and connect the device to PC.
• Click on uiautomator dump button
• Click on the text and you can copy the text from "Node details" on the side.

Incase of webview, the above approach might not work.

If you want to do it programmatically, do check at android.getText() method

Ics