Android-dev AudioRecord without blocking or threads

Question

I wish to record the microphone audio stream so I can do realtime DSP on it.

I want to do so without having to use threads and without having .read() block while it waits for new audio data.

UPDATE/ANSWER: It's a bug in Android. 4.2.2 still has the problem, but 5.01 IS FIXED! I'm not sure where the divide is but that's the story.

NOTE: Please don't say "Just use threads." Threads are fine but this isn't about them, and the android developers intended for AudioRecord to be fully usable without me having to specify threads and without me having to deal with blocking read(). Thank you!

Here is what I have found:

When the AudioRecord object is initialized, it creates its own internal ring type buffer. When .start() is called, it begins recording to said ring buffer (or whatever kind it really is.)

When .read() is called, it reads either half of bufferSize or the specified number of bytes (whichever is less) and then returns.

If there is more than enough audio samples in the internal buffer, then read() returns instantly with the data. If there is not enough yet, then read() waits till there is, then returns with the data.

.setRecordPositionUpdateListener() can be used to set a Listener, and .setPositionNotificationPeriod() and .setNotificationMarkerPosition() can be used to set the notification Period and Position, respectively.

However, the Listener seems to be never called unless certain requirements are met:

1: The Period or Position must be equal to bufferSize/2 or (bufferSize/2)-1.

2: A .read() must be called before the the Period or Position timer starts counting - in other words, after calling .start() then also call .read(), and each time the Listener is called, call .read() again.

3: .read() must read at least half of bufferSize each time.

So using these rules I am able to get the callback/Listener working, but for some reason the reads are still blocking and I can't figure out how to get the Listener to only be called when there is a full read's worth.

If I rig up a button view to click to read, then I can tap it and if tap rapidly, read blocks. But if I wait for the audio buffer to fill, then the first tap is instant (read returns right away) but subsiquent rapid taps are blocked because read() has to wait, I guess.

Greatly appreciated would be any insight on how I might make the Listener work as intended - in such a way that my listener gets called when there's enough data for read() to return instantly.

Below is the relavent parts of my code.

I have some log statements in my code which send strings to logcat which allows me to see how long each command is taking, and this is how I know that read() is blocking. (And the buttons in my simple test app also are very doggy slow to respond when it is reading repeatedly, but CPU is not pegged.)

Thanks, ~Jesse

In my OnCreate():

    bufferSize=AudioRecord.getMinBufferSize(samplerate,AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT)*4;
        recorder = new AudioRecord (AudioSource.MIC,samplerate,AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT,bufferSize);
        recorder.setRecordPositionUpdateListener(mRecordListener);
        recorder.setPositionNotificationPeriod(bufferSize/2);
        //recorder.setNotificationMarkerPosition(bufferSize/2);
        audioData = new short [bufferSize];

    recorder.startRecording();
            samplesread=recorder.read(audioData,0,bufferSize);//This triggers it to start doing the callback.

Then here is my listener:

public OnRecordPositionUpdateListener mRecordListener = new OnRecordPositionUpdateListener() 
{
    public void onPeriodicNotification(AudioRecord recorder) //This one gets called every period. 
    {
        Log.d("TimeTrack", "AAA");
        samplesread=recorder.read(audioData,0,bufferSize);
        Log.d("TimeTrack", "BBB");
        //player.write(audioData, 0, samplesread);
        //Log.d("TimeTrack", "CCC");
        reads++;
    }
    @Override
    public void onMarkerReached(AudioRecord recorder) //This one gets called only once -- when the marker is reached.
    {
        Log.d("TimeTrack", "AAA");
        samplesread=recorder.read(audioData,0,bufferSize);
        Log.d("TimeTrack", "BBB");
        //player.write(audioData, 0, samplesread);
        //Log.d("TimeTrack", "CCC");
    }
};

UPDATE: I have tried this on Android 2.2.3, 2.3.4, and now 4.0.3, and all act the same. Also: There is an open bug on code.google about it - one entry started in 2012 by someone else then one from 2013 started by me (I didn't know about the first):

UPDATE 2016: Ahhhh finally after years of wondering if it was me or android, I finally have answer! I tried my above code on 4.2.2 and same problem. I tried above code on 5.01, AND IT WORKS!!! And the initial .read() call is NOT needed anymore either. Now, once the .setPositionNotificationPeriod() and .StartRecording() are called, mRecordListener() just magically starts getting called every time there is data available now so it no longer blocks, because the callback is not called until after enough data has been recorded. I haven't listened to the data to know if it's recording correctly, but the callback is happening like it should, and it is not blocking the activity, like it used to!

http://code.google.com/p/android/issues/detail?id=53996

http://code.google.com/p/android/issues/detail?id=25138

If folks who care about this bug log in and vote for and/or comment on the bug maybe it'll get addressed sooner by Google.

Brittne

Answer 1

It's late answear, but I think I know where Jesse did a mistake. His read call is getting blocked because he is requesting shorts which are sized same as buffer size, but buffer size is in bytes and short contains 2 bytes. If we make short array to be same length as buffer we will read twice as much data.

The solution is to make audioData = new short[bufferSize/2] If the buffer size is 1000 bytes, this way we will request 500 shorts which are 1000 bytes.

Also he should change samplesread=recorder.read(audioData,0,bufferSize) to samplesread=recorder.read(audioData,0,audioData.length)

UPDATE

Ok, Jesse. I can see where another mistake can be - the positionNotificationPeriod. This value have to be large enought so it won't call the listener too often and we need to make sure that when the listener is called the bytes to read are ready to be collected. If bytes won't be ready when the listener is called, the main thread will get blocked by recorder.read(audioData, 0, audioData.length) call until requested bytes get's collected by AudioRecord. You should calculate buffer size and shorts array length based on time interval you set - how often you want the listener to be called. Position notification period, buffer size and shorts array length all have to be adjusted correctly. Let me show you an example:

int periodInFrames = sampleRate / 10;
int bufferSize = periodInFrames * 1 * 16 / 8;
audioData = new short [bufferSize / 2];

int minBufferSize = AudioRecord.getMinBufferSize(sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
if (bufferSize < minBufferSize) bufferSize = minBufferSize;

recorder = new AudioRecord(AudioSource.MIC, sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, buffersize);
recorder.setRecordPositionUpdateListener(mRecordListener);
recorder.setPositionNotificationPeriod(periodInFrames);
recorder.startRecording();

public OnRecordPositionUpdateListener mRecordListener = new OnRecordPositionUpdateListener() {
    public void onPeriodicNotification(AudioRecord recorder) {
        samplesread = recorder.read(audioData, 0, audioData.length);
        player.write(short2byte(audioData));
    }
};

private byte[] short2byte(short[] data) {
        int dataSize = data.length;
        byte[] bytes = new byte[dataSize * 2];

        for (int i = 0; i < dataSize; i++) {
            bytes[i * 2] = (byte) (data[i] & 0x00FF);
            bytes[(i * 2) + 1] = (byte) (data[i] >> 8);
            data[i] = 0;
        }
        return bytes;
    }

So now a bit of explanation.

First we set how often the listener have to be called to collect audio data (periodInFrames). PositionNotificationPeriod is expressed in frames. Sampling rate is expressed in frames per second, so for 44100 sampling rate we have 44100 frames per second. I divided it by 10 so the listener will be called every 4410 frames = 100 milliseconds - that's reasonable time interval.

Now we calculate buffer size based on our periodInFrames so any data won't be overriden before we collect it. Buffer size is expressed in bytes. Our time interval is 4410 frames, each frame contains 1 byte for mono or 2 bytes for stereo so we multiply it by number of channels (1 in your case). Each channel contains 1 byte for ENCODING_8BIT or 2 bytes for ENCODING_16BIT so we multiply it by bits per sample (16 for ENCODING_16BIT, 8 for ENCODING_8BIT) and divide it by 8.

Then we set audioData length to be half of the bufferSize so we make sure that when the listener gets called, bytes to read are already there waiting to be collected. That's because short contains 2 bytes and bufferSize is expressed in bytes.

Then we check if bufferSize is large enought to succesfully initialize AudioRecord object, if it's not then we set bufferSize to it's minimal size - we don't need to change our time interval or audioData length.

In our listener we read and store data to short array. That's why we use audioData.length instead buffer size, because only audioData.length can tell us the number of shorts the buffer contains.

I had it working some time ago so please let me know if it will work for you.

Oquendo

Thanks Szymon! Have you tried your fix on pre-kitkat? I kind of gave up on it and haven't tried recently. A post at the following link suggests the problem was fixed between 4.2.1 and 4.4.2, but I haven't tested it yet myself. I'll try to set up dev env soon and try your fix. code.google.com/p/android/issues/detail?id=53996 Thanks again. ~Jesse – Brittne

OK I tried your fix, but it didn't work. Also tried my app on Lollipop and that has same problem. But now that I think about it, audioData is just an array of shorts. Its size does not matter as long as it's big enough. I did try making it bufferSize/2 but that would not work at all because recorder.Read checks the size of it's target buffer and returns -2 if it's not big enough. If you do manage to actually get non-blocking reads going, I would be delighted to know how! Thanks, ~Jesse – Brittne

Answer 2

I'm not sure why you're avoiding spawning separate threads, but if it's because you don't want have to deal with coding them properly, you can use .schedule on a Timer object after each .read, where the time interval is set to the time it takes to get your buffer filled (number of samples in buffer / sampleRate). Yes I know this is using a separate thread, but this advice was given assuming that the reason you were avoiding using threads was to avoid having to code them properly.

This way, the longest time it can possibly block the thread for should be neglible. But I don't know why you'd want to do that.

If the above reason is not why you're avoiding using separate threads, may I ask why?

Also, what exactly do you mean by realtime? Do you intend to playback the affected audio using, let's say, an AudioTrack? Because the latency on most Android devices is pretty bad.

Huggermugger

I'll probably end up just spawning a thread. It seems Android is more buggy than a beehive and nothings getting fixed (25,000 New unassigned defect bugs, dating years back some of them..) In answer to your questions, I wanted to avoid threads because I wanted to avoid messy hacks and do it right, if I could. Looks like that won't happen, so next best is to use the cleanest hack - which is another thread. Threads are great - don't get me wrong. By realtime I mean oscilloscope app or spectrum analyzer, etc. Or playing audiotrk – Brittne