Android with Nexus 6 -- how to avoid decreased OpenSL audio thread priority relating to app focus?

Question

I'm encountering a strange problem when trying to implement low-latency streaming audio playback on a Nexus 6 running Android 6.0.1 using OpenSL ES.

My initial attempt seemed to be suffering from starvation issues, so I added some basic timing benchmarks in the buffer completion callback function. What I've found is that audio plays back fine if I continually tap the screen while my app is open, but if I leave it alone for a few seconds, the callback starts to take much longer. I'm able to reproduce this behavior consistently. A couple of things to note:

• "a few seconds" ~= 3-5 seconds, not long enough to trigger a screen change
• My application's activity sets FLAG_KEEP_SCREEN_ON, so no screen changes should occur anyway
• I have taken no action to try to increase the audio callback thread's priority, since I was under the impression that Android reserves high priority for these threads already
• The behavior occurs on my Nexus 6 (Android 6.0.1), but not on a Galaxy S6 I also have available (Android 5.1.1).

The symptoms I'm seeing really seem like the OS kicks down the audio thread priority after a few seconds of non-interaction with the phone. Is this right? Is there any way I can avoid this behavior?

Conjoint

Also doesn't occur on a Nexus 5 running Android 4.4.2. – Floret

Logging thread priority during playback seems to show that the thread priority isn't actually changing when tapping the screen. Maybe the touch event is triggering an interrupt that messes with what the scheduler is likely to get back to? – Floret

I'm having the same issue in my apps. As long as I interact with the screen, all runs smooth as it should. As soon as I stop touching the screen, it starts with massive drop outs after a few (around 3-5 seconds). When I start interacting with the screen again, it gets back to smooth operation. It doesn't matter if the screen interaction does anything real, nor if the finger is triggering with a specific control/button. As soon as a finger is moving on the screen, all audio runs smooth. It doesn't happen on Android 5.x, it just started with 6.0, and it's also the case on Android N. – Mingrelian

It is definitely not a performance problem in the app, I did tests with a single pre-calculated sine wave, the behavior is the same. I've used a nexus 6 for the tests. I also noticed that the behavior is exactly the same when the app goes to the background "for real". – Mingrelian

Answer 1

While watching the latest Google I/O 2016 audio presentation, I finally found the cause and the (ugly) solution for this problem.

Just watch the around one minute of this you tube clip (starting at 8m56s): https://youtu.be/F2ZDp-eNrh4?t=8m56s

It explains why this is happening and how you can get rid of it.

In fact, Android slows the CPU down after a few seconds of touch inactivity to reduce the battery usage. The guy in the video promises a proper solution for this soon, but for now the only way to get rid of it is to send fake touches (that's the official recommendation).

Instrumentation instr = new Instrumentation();
instr.sendKeyDownUpSync(KeyEvent.KEYCODE_BACKSLASH); // or whatever event you prefer

Repeat this with a timer every 1.5 seconds and the problem will vanish.

I know, this is an ugly hack, and it might have ugly side effects which must be handled. But for now, it is simply the only solution.

Update: Regarding your latest comment ... here's my solution. I'm using a regular MotionEvent.ACTION_DOWN at a location outside of the screen bounds. Everything else interfered in an unwanted way with the UI. To avoid the SecurityException, initialize the timer in the onStart() handler of the main activity and terminate it in the onStop() handler. There are still situations when the app goes to the background (depending on the CPU load) in which you might run into a SecurityException, therefore you must surround the fake touch call with a try catch block.

Please note, that I'm using my own timer framework, so you have to transform the code to use whatever timer you want to use.

Also, I cannot ensure yet that the code is 100% bulletproof. My apps have that hack applied, but are currently in beta state, therefore I cannot give you any guarantee if this is working correctly on all devices and Android versions.

Timer fakeTouchTimer = null;
Instrumentation instr;
void initFakeTouchTimer()
{
    if (this.fakeTouchTimer != null)
    {
        if (this.instr == null)
        {
            this.instr = new Instrumentation();
        }
        this.fakeTouchTimer.restart();
    }
    else
    {
        if (this.instr == null)
        {
            this.instr = new Instrumentation();
        }
        this.fakeTouchTimer = new Timer(1500, Thread.MIN_PRIORITY, new TimerTask()
        {
            @Override
            public void execute()
            {
                if (instr != null && fakeTouchTimer != null && hasWindowFocus())
                {
                    try
                    {
                        long downTime = SystemClock.uptimeMillis();

                        MotionEvent event = MotionEvent.obtain(downTime, downTime, MotionEvent.ACTION_DOWN, -100, -100, 0);
                        instr.sendPointerSync(event);
                        event.recycle();
                    }
                    catch (Exception e)
                    {
                    }
                }
            }
        }, true/*isInfinite*/);
    }
}
void killFakeTouchTimer()
{
    if (this.fakeTouchTimer != null)
    {
        this.fakeTouchTimer.interupt();
        this.fakeTouchTimer = null;
        this.instr = null;
    }
}

@Override
protected void onStop()
{
    killFakeTouchTimer();
    super.onStop();

    .....
}

@Override
protected void onStart()
{
    initFakeTouchTimer();
    super.onStart();

    .....
}

Mingrelian

Thanks for the answer; this definitely seems like the issue I was trying to get to the heart of. I've updated my app with their suggested work-around, but it seems unfortunately more hackish than anticipated. In particular, it can crash with a security exception if the volume warning comes up during the simulated touch event (no cross-app programmatic touch events). Have you had a chance to implement a more robust version of this yourself? – Floret

That's what I meant with ugly side effects. The SecurityException is just one of them. I've updated my answer with my final solution. – Mingrelian

I just noticed that I didn't take care of situations like the volume warning (or other situations of the same nature) so far. I updated the solution to have all those situations covered. – Mingrelian

Tried this, the fake motion events will interfere with actual users touches. Sending KEYCODE_BACKSLASH selects the first view in the layout, instead. :/ – Cosy

Yes, KEYCODE_BACKSLASH interferes with the UI, that's correct. But look at the final solution. It only fires MotionEvent.ACTION_DOWN, -100, -100, 0), that means an ACTION_DOWN event (not ACTION_UP) outside the screen bounds (-100, -100). These events cannot interfere, because they get dropped by the Android OS. But they keep the process at max speed. Also do only apply the fix for Nexus 6 (shamu) devices running on Android SDK level 23 or higher, since the problem occurs only in exact this situation. The fix is live in all our apps since more than two months and there are no problems at all. – Mingrelian

Thanks for the answer! Iv'e had this problem for years and the developers did not seem to care about it. Finally they at least noticed it :-) The "fun" thing is that they released the low latency audio support at Android 4.1/4.2 and the year after they introduced the agressive power savings feature that more or less undid the low latency audio support. At least for me that generating audio based on input from midi keyboards. – Anglophobe

Answer 2

It is well known that the audio pipeline in Android 6 has been completely rewritten. While this improved latency-related issues in most cases, it is not impossible that it generated a number of undesirable side-effects, as is usually the case with such large-scale changes.

While your issue does not seem to be a common one, there are a few things you might be able to try:

• Increase the audio thread priority. The default priority for audio threads in Android is -16, with the maximum being -20, usually only available to system services. While you can't assign this value to you audio thread, you can assign the next best thing: -19 by using the ANDROID_PRIORITY_URGENT_AUDIO flag when setting the thread's priority.

• Increase the number of buffers to prevent any kind of jitter or latency (you can even go up to 16). However on some devices the callback to fill a new buffer isn’t always called when it should.

• This SO post has several suggestions to improve audio latency on Anrdoid. Of particular interest are points 3, 4 and 5 in the accepted answer.

• Check whether the current Android system is low-latency-enabled by querying whether hasSystemFeature(FEATURE_AUDIO_LOW_LATENCY) or hasSystemFeature(FEATURE_AUDIO_PRO).

---

Additionally, this academic paper discusses strategies for improving audio latency-related issues in Android/OpenSL, including buffer- and callback interval-related approaches.

Algae

Thanks for the answer. Unfortunately, I've already read all of these (very helpful) resources and applied the techniques suggested therein. I'm running at the native device sample rate and at the device's native buffer size on a device that claims to support low-latency audio (in fact, I believe the Nexus 6 clocks in at the smallest native buffer yet at 192 frames -- 4 ms at 48kHz). Increasing buffer number hasn't helped for the reason described in the heatvst.com article you linked. My question is specifically about why UI interaction seems to affect audio playback. – Floret

Do you have any third-party apps installed that might gain focus in the background after a couple of seconds of non-interaction? – Algae

How does one make the audio thread ANDROID_PRIORITY_URGENT_AUDIO priority? – Bladderwort

Answer 3

Force resampling to native device sample rate on Android 6.

Use the device's native sample rate of 48000. For example:

SLDataFormat_PCM dataFormat;

dataFormat.samplesPerSec = 48000;

Futtock