Glass voice command nearest match from given list

Question

With Glass you can launch an app via the 'OK, Glass' menu and it seems to pick the nearest match unless a command is miles off, and you can obviously see the list of commands.
Is there anyway from within the app, or from the voice prompt (after the initial app trigger) to have a similar list given and return the nearest match.

Random (non-real world) example, an app that shows you a colour, "OK Glass, show the colour red"

'show the colour' could be your voice trigger and seems to be matched by glass on a 'nearest neighbor' method, however 'red' is just read in as free text and could be easily misheard as 'dread' or 'head', or even 'read' as there is no way of differentiating 'read' from 'red'.

Is there a way to pass a list of pre-approved option (red, green, blue, orange*, etc.) to this stage, or to another voice prompt within the app so the user can see the list and get more accurate results when there is a finite set of expected responses (like the main ok glass screen)?

*ok well nothing rhymes with orange, we're probably safe there

Edythedythe

Answer 1

The Google GDK doesn't support this feature yet. However, the necessary features are already available in some libraries and you can use them as long as the GDK doesn't support this natively. What you have to do:

1. Pull the GlassVoice.apk from your Glass: adb pull /system/app/GlassVoice.apk

2. Use dex2jar to convert this apk into a jar file.

3. Add the jar file to your build path

Now you can use this library like this:

public class VoiceActivity extends Activity {

    private VoiceInputHelper mVoiceInputHelper;
    private VoiceConfig mVoiceConfig;

        @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.voice_activity);

        String[] items = {"red", "green", "blue", "orange"};
        mVoiceConfig = new VoiceConfig("MyVoiceConfig", items);
        mVoiceInputHelper = new VoiceInputHelper(this, new MyVoiceListener(mVoiceConfig),
                VoiceInputHelper.newUserActivityObserver(this));
    }

    @Override
    protected void onResume() {
        super.onResume();
        mVoiceInputHelper.addVoiceServiceListener();
    }

    @Override
    protected void onPause() {
        super.onPause();
        mVoiceInputHelper.removeVoiceServiceListener();
    }

    public class MyVoiceListener implements VoiceListener {
        protected final VoiceConfig voiceConfig;

        public MyVoiceListener(VoiceConfig voiceConfig) {
            this.voiceConfig = voiceConfig;
        }

        @Override
        public void onVoiceServiceConnected() {
            mVoiceInputHelper.setVoiceConfig(mVoiceConfig, false);
        }

        @Override
        public void onVoiceServiceDisconnected() {

        }

        @Override
        public VoiceConfig onVoiceCommand(VoiceCommand vc) {
            String recognizedStr = vc.getLiteral();
            Log.i("VoiceActivity", "Recognized text: "+recognizedStr);

            return voiceConfig;
        }

        @Override
        public FormattingLogger getLogger() {
            return FormattingLoggers.getContextLogger();
        }

        @Override
        public boolean isRunning() {
            return true;
        }

        @Override
        public boolean onResampledAudioData(byte[] arg0, int arg1, int arg2) {
            return false;
        }

        @Override
        public boolean onVoiceAmplitudeChanged(double arg0) {
            return false;
        }

        @Override
        public void onVoiceConfigChanged(VoiceConfig arg0, boolean arg1) {

        }
    }

}

Jeffery

I tried it.. and it worked perfectly. Thank you so much! – Soloman

I just updated to XE16 and it doesn't seem to work anymore. After updating, I pulled the new apk from /system/private-app/GlassVoice.apk, used dex2jar to convert it, and added the jar to my build path. Ran the same code that worked perfectly on XE12. Now it seems that the VoiceListener never catches any VoiceCommand after the first one it detects. @Jeffery Do you have any idea why that is and how to fix it? – Sternum

Sorry, but I had to give away my Google Glass. I cannot check this anymore. – Jeffery

@Sternum any luck resolving this issue? I'm not even able to get the GlassVoice.apk from Glass as it doesn't exist... – Naturalism

Curious about this myself too. Looking to use it on XE 16.11. – Accrue

@Naturalism You can pull the Glassvoice.apk but it's now in /system/private-app instead of /system/app. For the issue I mentioned above, I found a way to hack around it but it's very messy and I'm not happy with it. The hack is to change the VoiceListener's VoiceConfig every time onVoiceCommand() is fired. – Sternum

the location seems to be /system/priv-app nowadays – Hysteria

@Sternum it seems like returning null instead of the config in the onVoiceCommand gives you the ability to keep recognizing commands one after another. – Hysteria

To find the latest location of the APK (cause it seems to change quite a bit), you can follow these commands: https://mcmap.net/q/63516/-how-do-i-get-an-apk-file-from-an-android-device – Hilar

I pulled the GlassVoice.apk from /system/priv-app and obtained the jar through dex2jar tool! But also returning null in the method VoiceListener.onVoiceCommand it seems to be unable to recognize anything. My glass are up to date with XE 17. – Soloman

Works for me on XE17, but only the first time I say something. – Rookery

When i am in an activity where i have given command "go" to move to another activity, but i am not saying "go" instead it takes any voice command and starts forcefully the next activity. – Matheny

It worked for me! But it comes another unexpected problem: "MyVoiceListener" only trigger once in my Activity. In this case, I need to trigger MyVoiceListener for many times. Any ideas? – Vesperal

I found the solution to my question - Simply return "null" in onVoiceCommand. And it works continuously :) Here's the ref: link – Vesperal

Will Google pull the app from the Glassware Gallery if it uses such private APIs? – Flake

Does anyone know why this functionality is missing from XE 22?com.google.glass.input.VoiceInputHelper and com.google.glass.input.VoiceListener are no longer in GlassVoice.apk. Have they moved, or has Google pulled the plug? – Chari

hi guys, I updated my Glass to XE22 and now, the library obtained from the pulled apk makes my application go in "GCC overhead exceeded limit".. Are you in the same situation or is it only a problem of mine? Thanks – Soloman

Answer 2

You can take advantage of the disambiguation step that occurs when multiple Activities or Services support the same Voice Trigger: simply have multiple Activities or Services in your application support "show me the color" as the voice trigger and label them with the color options.

Your manifest would look something like:

<application
        android:allowBackup="true"
        android:label="@string/app_name"
        android:icon="@drawable/icon_50"
        >

    <activity
            android:name="com.mycompany.RedActivity"
            android:label="@string/red"
            android:icon="@drawable/icon_red"
            >
        <intent-filter>
            <action android:name="com.google.android.glass.action.VOICE_TRIGGER"/>
        </intent-filter>
        <meta-data
                android:name="com.google.android.glass.VoiceTrigger"
                android:resource="@xml/activity_start"
                />
    </activity>

    <activity
            android:name="com.mycompany.BlueActivity"
            android:label="@string/blue"
            android:icon="@drawable/icon_blue"
            >
        <intent-filter>
            <action android:name="com.google.android.glass.action.VOICE_TRIGGER"/>
        </intent-filter>
        <meta-data
                android:name="com.google.android.glass.VoiceTrigger"
                android:resource="@xml/activity_start"
                />
    </activity>
    <!-- ... -->
</application>

Those Activities or Services would only be used as a "trampoline" to launch the main logic of your app with the color selection.

Herron

Thanks, this seems like an interesting work around, although feels pretty hacky and unmaintainable, especially if you have a long list of possible options, I don't suppose you know of any future plans to support this type of interaction a little more cleanly? – Edythedythe

@Herron are you sure this solution works? Because I'm in the same situation of Ben. So the recognizer which handles VoiceTrigger intents will automatically choose the nearest of those labels? I mean.. if it understands "Read" instead of "Red", will it automatically choose "Red" instead of "Blue" because "Red" would be recognized to be more similar to the "Read" word it previously understood? – Soloman

When using this approach, the recognizer will not be run but instead, the process used to launch an Activity/Service will trigger the correct one according to what it recognized. – Herron

Oh, ok. So it's only a tricky way to exclude wrongly-recognized speeches (hope I understood it right). I think @blueshadow's solution fits better Bob's problem, anyway you gave me a nice input: thanks man ;) – Soloman

Answer 3

If you haven't already, you should take a look at contextual voice menus that were added just a few weeks ago to the GDK. I had your exact same problem just the day before it was released, looking at it the next day and finding this helped me a lot! :)

Vasos

I just got familiar with this option, but there it is said, that commands which are not listed in VoiceTriggers.Command and ContextualMenus.Command can be used only in development mode. Maybe I missed something and I can use my own commands in production? – Urey

No, that's true. You won't be able to distribute your glassware through MyGlass with the development permission. It is however mentioned that you are able to submit your own commands for approval so you will be able to use them. – Vasos