Android Speech Recognition as a service on Android 4.1 & 4.2

Question

I have managed to get continuous speech recognition working (using the SpeechRecognizer class) as a service on all Android versions up to 4.1. My question concerns getting it working on versions 4.1 and 4.2 as it is known there is a problem in that the API doesn't do as documented in that a few seconds after voice recognition is started, if no voice input has been detected then it's as if the speech recogniser dies silently. (http://code.google.com/p/android/issues/detail?id=37883)

I have found a question which proposes a work-around to this problem (Voice Recognition stops listening after a few seconds), but I am unsure as how to implement the Handler required for this solution. I am aware of the 'beep' that will happen every few seconds that this workaround will cause, but getting continuous voice recognition is more important for me.

If anyone has any other alternative workarounds then I'd like to hear those too.

Lundy

On my Nexus S with android 4.1.1, the speech recognizer does not die but behaves differently than other phones with version 4.0. With 4.0 after a few second (say 5s), I got error no speech input. With 4.1.1 about at least 3 times longer than 4.0 (15s), I got error other network related errors. Thus if a user speak after say 5s, then the speech recognizer would not pick it up because it still processes the "no input error". In conclusion, in version 4.1.1 the "no speech input" is treated as "other network related errors" and it takes a lot longer for the server to return this error. – Basicity

There isn't a problem with 4.0 as up until the point when the onError() method is called, the speech recogniser is still active and I can simply 'restart' the voice recognition - allowing continuous recognition. I am aware of what happens on 4.1.1, but waiting for onError() to be called doesn't help me provide continuous recognition as there is a long delay between when the speech recogniser becomes inactive and waiting for onError() to be called. (Almost a minute in some cases!) – Lundy

As a work around you can implement a timer on onReadyForSpeech and after say 5s if onEndofSpeech has not been called, then call cancel and startListening again. – Basicity

I have tried this, and detecting the timeout is not a problem. The problem comes when starting the listener again. If you look into the Android source for the SpeechRecognizer class (github.com/android/platform_frameworks_base/blob/master/core/…) you will see that a checkIsCalledFromMainThread() method is called and I am unsure how to call startListening() from inside the main thread when a timing workaround is being used. – Lundy

I did this by sending a start listening message to my service message handler. – Basicity

That sounds like what I need to do then, could you explain in a little more detail how to do that? – Lundy

Did you solve the problem? Will you share the sample code with me? – Youthen

@ArdaKazancı - The answer to the problem is the marked answer below. – Lundy

Answer 1

This is a work around for android version 4.1.1.

public class MyService extends Service
{
    protected AudioManager mAudioManager; 
    protected SpeechRecognizer mSpeechRecognizer;
    protected Intent mSpeechRecognizerIntent;
    protected final Messenger mServerMessenger = new Messenger(new IncomingHandler(this));

    protected boolean mIsListening;
    protected volatile boolean mIsCountDownOn;
    private boolean mIsStreamSolo;

    static final int MSG_RECOGNIZER_START_LISTENING = 1;
    static final int MSG_RECOGNIZER_CANCEL = 2;

    @Override
    public void onCreate()
    {
        super.onCreate();
        mAudioManager = (AudioManager) getSystemService(Context.AUDIO_SERVICE); 
        mSpeechRecognizer = SpeechRecognizer.createSpeechRecognizer(this);
        mSpeechRecognizer.setRecognitionListener(new SpeechRecognitionListener());
        mSpeechRecognizerIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
        mSpeechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                                         RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
        mSpeechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE,
                                         this.getPackageName());
    }

    protected static class IncomingHandler extends Handler
    {
        private WeakReference<MyService> mtarget;

        IncomingHandler(MyService target)
        {
            mtarget = new WeakReference<MyService>(target);
        }


        @Override
        public void handleMessage(Message msg)
        {
            final MyService target = mtarget.get();

            switch (msg.what)
            {
                case MSG_RECOGNIZER_START_LISTENING:

                    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN)
                    {
                        // turn off beep sound  
                        if (!mIsStreamSolo)
                        {
                            mAudioManager.setStreamSolo(AudioManager.STREAM_VOICE_CALL, true);
                            mIsStreamSolo = true;
                        }
                    }
                     if (!target.mIsListening)
                     {
                         target.mSpeechRecognizer.startListening(target.mSpeechRecognizerIntent);
                         target.mIsListening = true;
                        //Log.d(TAG, "message start listening"); //$NON-NLS-1$
                     }
                     break;

                 case MSG_RECOGNIZER_CANCEL:
                    if (mIsStreamSolo)
                   {
                        mAudioManager.setStreamSolo(AudioManager.STREAM_VOICE_CALL, false);
                        mIsStreamSolo = false;
                   }
                      target.mSpeechRecognizer.cancel();
                      target.mIsListening = false;
                      //Log.d(TAG, "message canceled recognizer"); //$NON-NLS-1$
                      break;
             }
       } 
    } 

    // Count down timer for Jelly Bean work around
    protected CountDownTimer mNoSpeechCountDown = new CountDownTimer(5000, 5000)
    {

        @Override
        public void onTick(long millisUntilFinished)
        {
            // TODO Auto-generated method stub

        }

        @Override
        public void onFinish()
        {
            mIsCountDownOn = false;
            Message message = Message.obtain(null, MSG_RECOGNIZER_CANCEL);
            try
            {
                mServerMessenger.send(message);
                message = Message.obtain(null, MSG_RECOGNIZER_START_LISTENING);
                mServerMessenger.send(message);
            }
            catch (RemoteException e)
            {

            }
        }
    };

    @Override
    public void onDestroy()
    {
        super.onDestroy();

        if (mIsCountDownOn)
        {
            mNoSpeechCountDown.cancel();
        }
        if (mSpeechRecognizer != null)
        {
            mSpeechRecognizer.destroy();
        }
    }

    protected class SpeechRecognitionListener implements RecognitionListener
    {

        @Override
        public void onBeginningOfSpeech()
        {
            // speech input will be processed, so there is no need for count down anymore
            if (mIsCountDownOn)
            {
                mIsCountDownOn = false;
                mNoSpeechCountDown.cancel();
            }               
            //Log.d(TAG, "onBeginingOfSpeech"); //$NON-NLS-1$
        }

        @Override
        public void onBufferReceived(byte[] buffer)
        {

        }

        @Override
        public void onEndOfSpeech()
        {
            //Log.d(TAG, "onEndOfSpeech"); //$NON-NLS-1$
         }

        @Override
        public void onError(int error)
        {
            if (mIsCountDownOn)
            {
                mIsCountDownOn = false;
                mNoSpeechCountDown.cancel();
            }
             mIsListening = false;
             Message message = Message.obtain(null, MSG_RECOGNIZER_START_LISTENING);
             try
             {
                    mServerMessenger.send(message);
             }
             catch (RemoteException e)
             {

             }
            //Log.d(TAG, "error = " + error); //$NON-NLS-1$
        }

        @Override
        public void onEvent(int eventType, Bundle params)
        {

        }

        @Override
        public void onPartialResults(Bundle partialResults)
        {

        }

        @Override
        public void onReadyForSpeech(Bundle params)
        {
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN)
            {
                mIsCountDownOn = true;
                mNoSpeechCountDown.start();

            }
            Log.d(TAG, "onReadyForSpeech"); //$NON-NLS-1$
        }

        @Override
        public void onResults(Bundle results)
        {
            //Log.d(TAG, "onResults"); //$NON-NLS-1$

        }

        @Override
        public void onRmsChanged(float rmsdB)
        {

        }

    }
}

02/16/2013 - Fix beep sound if you use Text To Speech in your app make sure to turn off Solo stream in onResults

Basicity

This seems to work, though I did not implement the mute methods inside of a service, but elsewhere. – Gagliardi

I wrote and tested the countdown timer as well as muting the beep sound yesterday. It work fine, however there may be a need to synchronize mIsCountDownOn instead of just declaring it volatile. – Basicity

I'm running this on a 4.1.2 just by creating a Main Activity that has a button that starts the service, but out of all the different methods in the Service file, only onCreate() and IncomingHandler(MyService target) are actually hit. And then nothing happens. Is this part of the 4.1 and 4.2 problem? – Doe

You have to send a START LISTENING message say either onCreate or in onStartCommand – Basicity

Add the setRecognitionListener in onCreate. – Basicity

@HoanNguyen Hi, I'm new to services. How would I start this service from an Activity (on a button click, for example)? Could you give a simple code snippet? Thanks! – Heavensent

startService(YourClass.this, MyService.class); – Basicity

@HoanNguyen I see, and would that automatically trigger the handler to start listening? – Heavensent

No, you can override onStartCommand and send a MSG_RECOGNIZER_START_LISTENING message – Basicity

For some reason, I get target.mSpeechRecognizerIntent and target.mAudioManager are null (causing exceptions) inside the incoming handler. Has anyone encountered this? The onCreate function definitely runs first and instantiates them, but when target is created, those objects are null. – Clerissa

Did you copy the exact code? If so, check if mAudioManager and mSpeechRecognizerIntent are null in onCreate. – Basicity

Yes I copied the code exactly, but Eclipse required that I add the unimplemented onBind method. mAudioManager and mSpeechRecognizerIntent are NOT null in onCreate. Everything seems to work fine until I get a null exception trying to access mAudioManager. – Clerissa

If target.mAudioManager is null then it should be null from the beginning. Check to see if target is null. – Basicity

target is not null but all the objects inside it are. – Clerissa

That is pretty strange, you should post it as a question with all the relevant codes. – Basicity

Thanks for your help. Posted here: #18039929 – Clerissa

@HoanNguyen Hi .I wanted to know if it is possible to use your service as continuous recognition until user press stop button .and from where you got such timing 5000 for countdown . will it make recognition skip some phrases that user uttered – Ducky

Yes you can by stopping your service. I used 5 seconds because I tested a little and before 5 seconds JB won't cause error. After 5 seconds I call cancel and startListening right away, so unless the user starts speaking in the microseconds gap there should be no problem. – Basicity

I cannot get this code to start listening for speech. onCreate() in the service is called, but onReadyForSpeech() is never called. How exactly do you get speech recognition to start listening?...I tried mSpeechRecognizer.startListening() in onCreate() & also overrided onStart and tried mSpeechRecognizer.startListening() there but speech recognition never starts. – Peti

Override onStartCommand and send a MSG_RECOGNIZER_START_LISTENING message – Basicity

I tried that like so: Message msg = new Message(); msg.what = MSG_RECOGNIZER_START_LISTENING; mServerMessenger.send(msg); I'm calling this service from an activity. – Peti

Ok Here it is: #18606013 – Peti

Anyone run into slow speech response time on <= ICS? See here: #21425223 – Peti

Is there a way to make it start listening again after the speech is recognized ? – Aristocratic

Send MSG_RECOGNIZER_START_LISTENING after you handled the result. – Basicity

I had to throw the mServerMessenger.send(msg) into a try/catch. @HoanNguyen, thanks for posting this btw... but have you any insight on what could be causing the mAudioManager to not mute on 4.1.2? I've put a breakpoint and seen that it isn't null, but alas, the beep sound. – Neckpiece

Found my own answer. If anyone can't get the audio muted, you might have success with this: target.mAudioManager.setStreamMute(AudioManager.STREAM_MUSIC, true); – Neckpiece

@Neckpiece that's still wrong because the whole audio system of the device would go off!! – Aristocratic

@Pedrum, that's true that all sound would be muted, but in all serious... saying "that's wrong" doesn't make my LG Optimus G respect any other 'beep' mute technique. – Neckpiece

@HoanNguyen.. still didn't prevent the beep for me. – Neckpiece

there's always a better way in programming.. ill keep tryinnn :) – Aristocratic

Is this supposed to work on Android < 4.1.1? I am having issues – Ziska

HI Please send demo code, how to use it services in activity and how to perform action regarding text. I am new in Android development. – Enwreathe

I believe the use of WeakReference in this code is flawed. A call to "get" on a WeakReference will return null if the object is garbage collected. The rationale for using a WeakReference in this case is also (imho) unclear and makes the example less readable. – Gheber

can you please me out with getting this service working? i have posted relevant questions and code here : #26322469 – Cartwright

First of all, it is a nice workaround. I know this post is old... I am having an issue with my media player being muted when the Speech Recognition service has started... Is there anyway I can avoid it muting my media player? Thanks – Coquillage

Instead of setStreamSolo try setStreamMute. – Basicity

I just tried mAudioManager.setStreamMute(AudioManager.STREAM_VOICE_CALL, true); but it didn't mute the beep. My GN2 is on 4.1.1 and I have GS3 is on 4.1.2. I recently updated Google service so that I can downloaded the offline Speech Recognition. Thanks – Coquillage

Try some other streams like STREAM_SYSTEM – Basicity

Ive tried all STREAM_... but none of them worked except STREAM_MUSIC which also mutes my media player... STREAM_SYSTEM was used to work before I updated Google service for Offline Speech Recognition. Google has changed STREAM_SYSTEM to STREAM_MUSIC for beep sound for some odd reasons... Way to go Google! – Coquillage

I don't think there is any thing you can do about this. – Basicity

@HoanNguyen - doesn't this answer result in a gap of around 200-500 mili seconds of no-listening (between one end of speech to one ready for speech)? – Veteran

What do you mean by this answer? The mute sound or the continuous speech recognizer in a service? – Basicity

@HoanNguyen - continuous speech recognizer in a service. – Veteran

Yes but I process the result and then do something else before I call start listening again. I also prompt the user to speak only when onReadyForSpeech is called, so I have no problem at all. – Basicity

It should work when screen if off. Change isStreamSolo to target.isStreamSolo in the IncomingHandler class. However turning off the beep does not work anymore. – Basicity

I am trying to do a simple keyword spotting in my service, so that I can listen for the word "hello". Unfortunately it is not working. Please help me: #35389220 – Pyelography

@HoanNguyen I am always getting Error in SpeechRecognitionListener. Error - 9. Can you help me resolving the same. Have debugged at least 20times, but nothing has helped me yet. – Marchall

@HoanNguyen Thanks for the same. I am in a strange issue this time, that when I don't anything for 5-10 secs, my and then say something for 11th sec or nth sec, it never listens to me. Why ? I mean after onReadyForSpeech is not followed by onBeginning in that case. – Marchall

If I say something in starting e.g when the service is just started, it listens to me well and I can process the results. I want to make it to listen endlessly or at least for some minutes. But it waits only for few seconds – Marchall

Answer 2

If you really want to implement continuous listening without internet connection you need to consider third-party packages, one of them is CMUSphinx, check Pocketsphinx android demo for example how to listen for keyword efficiently in offline and react on the specific commands like a key phrase "oh mighty computer". The code to do that is simple:

you create a recognizer and just add keyword spotting search:

recognizer = defaultSetup()
        .setAcousticModel(new File(modelsDir, "hmm/en-us-semi"))
        .setDictionary(new File(modelsDir, "lm/cmu07a.dic"))
        .setKeywordThreshold(1e-5f)
        .getRecognizer();

recognizer.addListener(this);
recognizer.addKeywordSearch(KWS_SEARCH_NAME, KEYPHRASE);
switchSearch(KWS_SEARCH_NAME);

and define a listener:

@Override
public void onPartialResult(Hypothesis hypothesis) {
    String text = hypothesis.getHypstr();
    if (text.equals(KEYPHRASE))
      //  do something
} 

Heraclitus

Sorry but in this way what exactly happen? You can decide a keyword as "hello" and then in the onPartialResult only if i say the keyword the tts starts? – Mcginn

Will it work even if the screen is off? Also, is KEYPHRASE a string? – Pyelography

1) yes, it works when screen is off, though you need to add wakeup lock to prevent your phone from sleep. 2) yes, it is arbitrary string. – Heraclitus

Hey I am trying to do a simple keyword spotting in my service, so that I can listen for the word "hello". Unfortunately it is not working. Please help me: #35389220 – Pyelography

@NikolayShmyrev. may i ask u answer this question - https://mcmap.net/q/339373/-doing-actions-after-sound-recognization-in-android/4568864 - thanks – Whisler

how can i set KEYWORD from edit text that user set it? – Whisler

Answer 3

For any of you who are trying to silence the beep sound, regrading the @HoanNguyen answer which is very good but be careful as said in the api set setStreamSolo is cumulative so if there is in error in the speech recognition and on error is called(for example no internet connection) then setStremSolo true is called again and again which will result in your app silencing the whole phone (very bad)! the solution to that is to add the setStremMute(false) to the speechRecognizer onError.

Ammoniacal

Very good catch. I implements a cancel() function which I called in onError which among other thing unset the stream solo. So I never have problem. – Basicity

Answer 4

check out my demo app : https://github.com/galrom/ContinuesVoiceRecognition

I recommand to use both PockeySphix and SpeechRecognizer.

Interspace

Hey I am trying to do a simple keyword spotting in my service, so that I can listen for the word "hello". Unfortunately it is not working. Please help me: #35389220 – Pyelography

@RuchirBaronia , have you checked my Github Project? Try to compare and see what is missing on your project. – Interspace

Hi Gal Rom, I actually did see your Github project, but I wasn't able to find out what was wrong in my project. Here is my stack trace. For some reason, I am not able to simply recognize for the word "hello". :( I would really appreciate any help, you seem to be the speech recognition expert! – Pyelography

@RuchirBaronia The only way I can help you is by seeing the Project. Feel Free to send it to [email protected] and I will try to see what's up there. – Interspace

@gal-rom your github code works! would it be possible for you to code in your github such that it can record audio in background (even when the screen is off, or the app is killed)? Something like this: fabcirablog.weebly.com/blog/… – Hadleigh

@GalRom thx a lot! I found a bug though: In onDestroy of SpeechRecognizerManager you check wether mIsStreamSolo is false, but you need to check if it is true, for reestablishing volume-settings! – Gasiform