Android unable read window content on few devices using accessibility service
Asked Answered
M

3

8

My requirement: Reading the text from pop up, dialog etc for particular app.

I have implemented an accessibility service and I am receiving proper events and data as per my requirement. However while testing I realized that on some devices instead of using a AlertDialog or Dialog they have used an activity(themed as a dialog). Hence in my accessibility event I receive only the activity title, is there a way I could find the text displayed by this particular pop up activity?

I have searched pretty hard but could not get much help on the topic nor did the documentation be of any good in this concern. There is not much in the code of accessibility service but if you still need I will post it later.

Thanks

Marvellamarvellous answered 21/12, 2015 at 7:51 Comment(9)
I would discourage this. While there are ways to access all of the information. If you're in random third party apps, there is no reliable way to identify between an Activity being used as an AlertDialog, and just an Activity. Forcing this behavior just to fix a random scenario in which developers decided to do weird things is actually introducing compatibility issues. Essentially the equivalent of a user agent violation in the Web Accessibility world.Specify
@ChrisCM: I agree with you. However in this case its not true, its not some ramdom scenario. It seems to be occuring with mediatek sdk since they have used "AlertActivity" instead of android default "AlertDialog". I am developing an app with such a feature I would like to support all devices and if I ignore it I will be losing potential market.Marvellamarvellous
In Android accessibility, sometimes supporting a fringe scenario means worse behavior under the expected scenario. It is unfortunate, but it is true. I'm fairly convinced this is one of those cases. The Accessibility APIs simply don't contain the information you need.Specify
Did you get any solution for this? I am also stuck with this issue.Spokeshave
No. I have not found a solution for this issue. As per my research it is a dialog within an activity (UssdAlertActivity) which is why it returns title "phone" in event.getText() as mentioned by Kunwar Avanish.Marvellamarvellous
what i have noticed that this issue comes on devies supporting meditek.Spokeshave
Correct observation, I meant to add that earlier. This UssdAlertActivity belongs to mediatek sdk which returns "phone" instead of dialog text. I wonder how accessibility services like Talkback work in such a case.Marvellamarvellous
Did you also faced issue in performGlobalAction(GLOBAL_ACTION_BACK) it was working fine for normal ussd response with message but response with an option to enter chioces for user were not responding on global action back?Spokeshave
Yes, GLOBAL_ACTION_BACK works only in case the dialog is cancelable. But most dialogs leave an option for further choices like you said and are not cancelable. So this does not work anymore. Later I found an alternative of sending a broadcast with intent cancel system dialog which also does not seem to work. As of now I have given up this work as there is no solution or work around I could find.Marvellamarvellous
S
5

Use accessiblityNodeInfo to get information even in the case of when phone is returning it will fetch the ussd response also it will dismiss dialog when there is option to enter multiple choices.

First of all in case of [pohne] event class name is returned as ussdalertactivty so i used only "alert" for identification of alert dialog of ussd response.

public void onAccessibilityEvent(AccessibilityEvent event) {
    if (event.getPackageName().toString().equals("com.android.phone")
                        && event.getClassName().toString().toLowerCase()
                                .contains("alert")) {

                    AccessibilityNodeInfo source = event.getSource();

                    if (source != null) {
                    String pcnResponse = fetchResponse(source);
                    }
    }

Now i have made a function called fetchResponse which will return the response from pcn as string and will also dismiss the dialog so need to do performGlobalAction(GLOBAL_ACTION_BACK).

private String fetchResponse(AccessibilityNodeInfo accessibilityNodeInfo) {

        String fetchedResponse = "";
        if (accessibilityNodeInfo != null) {
            for (int i = 0; i < accessibilityNodeInfo.getChildCount(); i++) {
                AccessibilityNodeInfo child = accessibilityNodeInfo.getChild(i);
                if (child != null) {

                    CharSequence text = child.getText();

                    if (text != null
                            && child.getClassName().equals(
                                    Button.class.getName())) {

                        // dismiss dialog by performing action click in normal
                        // cases
                        if((text.toString().toLowerCase().equals("ok") || text
                                        .toString().toLowerCase()
                                        .equals("cancel"))) {

                            child.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                            return fetchedResponse;

                        }

                    } else if (text != null
                            && child.getClassName().equals(
                                    TextView.class.getName())) {

                        // response of normal cases
                        if (text.toString().length() > 10) {
                            fetchedResponse = text.toString();
                        }

                    } else if (child.getClassName().equals(
                            ScrollView.class.getName())) {

                        // when response comes as phone then response can be
                        // retrived from subchild
                        for (int j = 0; j < child.getChildCount(); j++) {

                            AccessibilityNodeInfo subChild = child.getChild(j);
                            CharSequence subText = subChild.getText();

                            if (subText != null
                                    && subChild.getClassName().equals(
                                            TextView.class.getName())) {

                                // response of different cases
                                if (subText.toString().length() > 10) {
                                    fetchedResponse = subText.toString();
                                }

                            }

                            else if (subText != null
                                    && subChild.getClassName().equals(
                                            Button.class.getName())) {

                                // dismiss dialog by performing action click in
                                // different
                                // cases
                                if ((subText.toString().toLowerCase()
                                                .equals("ok") || subText
                                                .toString().toLowerCase()
                                                .equals("cancel"))) {
                                    subChild.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                                    return fetchedResponse;
                                }

                            }
                        }
                    }
                }
            }
        }
        return fetchedResponse;
    }

Hope this will solve the issue irrespective of the devices.

Spokeshave answered 26/4, 2016 at 10:2 Comment(3)
Thanks Vishal. I do not possess a mediatek device as of now let me check this out once I get my hands on it.Marvellamarvellous
Hi! What is a variable isUSSDFromApp?Abseil
@V.Kalyuzhnyu sorry "issUssdFromApp" I have kept this for my own task. you can ignore that and i have edited that in my answer as well.Spokeshave
L
3

This is the code I use and it works for me:

public class USSDService extends AccessibilityService {

public static String TAG = USSDService.class.getSimpleName();

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
    Log.d(TAG, "onAccessibilityEvent");

    AccessibilityNodeInfo source = event.getSource();
    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED && !String.valueOf(event.getClassName()).contains("AlertDialog")) {
        return;
    }
    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED && (source == null || !source.getClassName().equals("android.widget.TextView"))) {
        return;
    }
    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED && TextUtils.isEmpty(source.getText())) {
        return;
    }

    List<CharSequence> eventText;

    if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED) {
        eventText = event.getText();
    } else {
        eventText = Collections.singletonList(source.getText());
    }

    String text = processUSSDText(eventText);

    if( TextUtils.isEmpty(text) ) return;

    // Close dialog
    performGlobalAction(GLOBAL_ACTION_BACK); // This works on 4.1+ only

    Log.d(TAG, text);
    // Handle USSD response here

}

private String processUSSDText(List<CharSequence> eventText) {
    for (CharSequence s : eventText) {
        String text = String.valueOf(s);
        // Return text if text is the expected ussd response
        if( true ) {
            return text;
        }
    }
    return null;
}

@Override
public void onInterrupt() {
}

@Override
protected void onServiceConnected() {
    super.onServiceConnected();
    Log.d(TAG, "onServiceConnected");
    AccessibilityServiceInfo info = new AccessibilityServiceInfo();
    info.flags = AccessibilityServiceInfo.DEFAULT;
    info.packageNames = new String[]{"com.android.phone"};
    info.eventTypes = AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED | AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED;
    info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
    setServiceInfo(info);
}
}

Declare it in Android manifest

<service android:name=".USSDService"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
<intent-filter>
    <action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data android:name="android.accessibilityservice"
    android:resource="@xml/ussd_service" />

Create a xml file that describes the accessibility service called ussd_service

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
android:accessibilityFeedbackType="feedbackGeneric"
android:accessibilityFlags="flagDefault"
android:canRetrieveWindowContent="true"
android:description="@string/accessibility_service_description"
android:notificationTimeout="0"
android:packageNames="com.android.phone" />
Leda answered 21/12, 2015 at 16:56 Comment(8)
This is still not helping out for micromax, lenovo devicesAmphiprostyle
@HenBoy331: Thanks for your code. Let me try this out.Marvellamarvellous
I am not geting ussd dialog message using event.gettext(); i m geting "PHONE" .. need to help for original message of ussd dialog.Catacomb
@KunwarAvanish what are you getting as parameter in processUSSDText method?Leda
eventText = event.getText(); ... same as like your code... and try many ways .. plz help.Catacomb
i am getting processUSSDText: [Phone] text only.. not geting alertdialog message.Catacomb
I am able to capture the incoming ussd message, but how to compare the incoming ussd message in case of dual sim phone? means if i receive a ussd message alert then how i can know that incoming ussd message is for which sim?Backwards
Worked fine for me. Don't forget to turn Accessibility on for you app in Settings->Device->Accessibility->"scroll down to see your app"Profluent
T
1

you should check whether there are any sub child for child nodes.

private void clickPerform(AccessibilityNodeInfo nodeInfo)
 {
   if(nodeInfo != null)
 {

            for (int i = 0; i < nodeInfo.getChildCount(); i++) {
                AccessibilityNodeInfo childNode = nodeInfo.getChild(i);
                Log.e("test", "clickPerform: "+childNode );
                if (childNode != null) {
                    for (int j = 0; j <= childNode.getChildCount(); j++) {
                        AccessibilityNodeInfo subChild = childNode.getChild(i);

                        if (String.valueOf(subChild.getText()).toLowerCase().equals("ok")) {
                            subChild.performAction(AccessibilityNodeInfo.ACTION_CLICK);
                        } else {
                            Log.e("t2", "clickPerform: ");
                        }
                    }
                }

                }
            }
}
Theocentric answered 17/5, 2018 at 5:20 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.