Android AudioRecord class - process live mic audio quickly, set up callback function

Question

I want to record audio from the mic and access it for possible playback in near real-time. I am unsure of how to use the Android AudioRecord class to record some mic audio and quickly access it.

For the AudioRecord class, the official site says 'the app polls the AudioRecord object in time', and 'the size of the buffer being filled determines the time-length of the recording before over-running unread data'. Later it's suggested that a larger buffer should be used when polling less frequently. They never actually show an example in code.

One example I've seen in a book uses the AudioRecord class to continuously read a buffer freshly populated with live mic audio, and then the app writes this data to an SD file. The pseudo-code looks something like -

set up AudioRecord object with buffer size and recording format info
set up a file and an output stream
myAudioRecord.startRecording();
while(isRecording)
{
    // myBuffer is being filled with fresh audio
    read audio data into myBuffer
    send contents of myBuffer to SD file
}
myAudioRecord.stop();

How this code synchronizes its reading with the rate of recording is unclear - is the boolean "isRecording" sequenced on and off properly elsewhere? It seems this code could either read too frequently or too infrequently, depending on how long the reading and writing takes.

The site doc also says the AudioRecord class has a nested class named OnRecordPositionUpdateListener which is defined as an interface. The information suggests that somehow, you specify the period you want for being notified of the progress of the recording, and the name of your event handler, and a call is automatically made to your event handler at the specified frequency. I think the structure, in pseudo-code would be something like -

set target of period update message = myListener
set period to be about every 250 ms
other code

myListener()
{
    if(record button was recently tapped)
        handle message that another 250 ms of fresh audio is available
        ie, read it and send it somewhere
)

I need to find some specific code which allows me to capture and process mic audio with a delay of less than about 500 ms. Android offers another class called MediaRecorder, but it doesn't support streaming, and I may want to stream live mic audio over a Wi-Fi network in near real-time. Where can I find some specific examples?

Bechtold

Did you ever solve this? I am thinking about adding a bounty to your question... – Cadelle

Bounty added.. no help so far – Cadelle

This question enlightened me a fair bit on the usage of the callbacks. – Permittivity

Answer 1

After experimenting lots with the notifications and a bunch of other techniques I settled on this code:

private class AudioIn extends Thread { 
     private boolean stopped    = false;

     private AudioIn() { 

             start();
          }

     @Override
     public void run() { 
            android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);
            AudioRecord recorder = null;
            short[][]   buffers  = new short[256][160];
            int         ix       = 0;

            try { // ... initialise

                  int N = AudioRecord.getMinBufferSize(8000,AudioFormat.CHANNEL_IN_MONO,AudioFormat.ENCODING_PCM_16BIT);

                   recorder = new AudioRecord(AudioSource.MIC,
                                              8000,
                                              AudioFormat.CHANNEL_IN_MONO,
                                              AudioFormat.ENCODING_PCM_16BIT,
                                              N*10);

                   recorder.startRecording();

                   // ... loop

                   while(!stopped) { 
                      short[] buffer = buffers[ix++ % buffers.length];

                      N = recorder.read(buffer,0,buffer.length);
                      //process is what you will do with the data...not defined here
                      process(buffer);
                  }
             } catch(Throwable x) { 
               Log.w(TAG,"Error reading voice audio",x);
             } finally { 
               close();
             }
         }

      private void close() { 
          stopped = true;
        }

    }

So far it's working pretty robustly on the half a dozen Android phones I've tried it on.

Hydrant

How do you properly handle buffer size? You need to create the smallest buffer size that does not result in buffer overflows on the device, so that you can process incoming audio ASAP. – Cadelle

Basically experimentation - I happened to be using a GSM codec so the 'chunk' size is a short[160]. AudioRecord.read(buffer,0,buffer.length) where buffer is short[160] returns as soon as a 'chunk' is ready. And experimenting with the number of internal AudioRecord buffers showed that 10x the minimum buffer size was sufficient to (mostly) avoid buffer overflows (and the application was tolerant of the occasionally dropped buffer). Hope that helps :-). – Hydrant

Well, you cannot experiment manually for every device, so you need some system in place to dynamically update the buffer size, right? I currently did it by implementing a maximum precision time, which checks if the read doesn't return too soon. If it returns sooner than 250ms, it'll increase buffer size. This works since the read is a blocking method. Not sure if there is a better way though. – Cadelle

Oh ... just realised I misread your first comment. The process(buffer) method simply sticks the read buffer into a queue for processing by a separate thread. So read(...) is always 'ready to read'. I didn't have much success with the notifications - you might have better luck. – Hydrant

You can use getMinBufferSize to get the minimum buffer size for a specific format. – Ratliff

In your code, you don't use (and need actually) 256 buffers of 160 short. Because you are processing one by one, you can use always the same. – Acrylyl

@Acrylyl In the real application the buffers were put onto a queue for processing by a thread and recycled after being processed – Hydrant

How do you get away with so low a sample rate (8000Hz)? It seems inutilable since it puts the Nyquist frequency well below the high end of human audition. – Goethite

The standard voice baseband (in telephony) is 0.3KHz-3.4 KHz - it's a legacy from the early days of analog telephony when it was decided that that was sufficient for intelligible conversation. Lots of google information if you're interested. – Hydrant

Answer 2

I wonder if you could combine these answers in the following way...

Use setPositionNotificationPeriod(160) before the while loop. This should cause the callback to be called every time 160 frames are read. Instead of calling process(buffer) inside of the thread that's doing the read loop, call process(buffer) from the callback. Use a variable to keep track of the last read buffer so you process the right one. As it is now, you block on the read, then you're not reading while you're processing. I think it might be better to separate those two.

Culinary

Why 160 ? what it should be in my case ... #9414498 ? – Enthusiasm

Answer 3

Here is the code you need to use the OnRecordPositionUpdateListener and Notification Period.

I noticed that in practice it does not send the notification consistently at the same exact time, I want, but it is close enough.

About detectAfterEvery:

The size of detectEvery needs to be large enough to hold just the amount of data you want. So for this example, we have a sample rate of 44100 Hz, that means we want 44100 samples per second. By setting the setPositionNotificationPeriod to be 44100, the code tells Android to callback after it has recorded 44100 samples, which is about every 1 second.

The complete code is here:

        final int sampleRate = 44100;
        int bufferSize =
                AudioRecord.getMinBufferSize(sampleRate,
                        AudioFormat.CHANNEL_CONFIGURATION_MONO,
                        AudioFormat.ENCODING_PCM_16BIT);

//aim for 1 second
        int detectAfterEvery = (int)((float)sampleRate * 1.0f);

        if (detectAfterEvery > bufferSize)
        {
            Log.w(TAG, "Increasing buffer to hold enough samples " + detectAfterEvery + " was: " + bufferSize);
            bufferSize = detectAfterEvery;
        }

        recorder =
                new AudioRecord(AudioSource.MIC, sampleRate,
                        AudioFormat.CHANNEL_CONFIGURATION_MONO,
                        AudioFormat.ENCODING_PCM_16BIT, bufferSize);
        recorder.setPositionNotificationPeriod(detectAfterEvery);

        final short[] audioData = new short[bufferSize];
        final int finalBufferSize = bufferSize;

        OnRecordPositionUpdateListener positionUpdater = new OnRecordPositionUpdateListener()
        {
            @Override
            public void onPeriodicNotification(AudioRecord recorder)
            {
                Date d = new Date();
//it should be every 1 second, but it is actually, "about every 1 second"
//like 1073, 919, 1001, 1185, 1204 milliseconds of time.
                Log.d(TAG, "periodic notification " + d.toLocaleString() + " mili " + d.getTime());
                recorder.read(audioData, 0, finalBufferSize);

                //do something amazing with audio data
            }

            @Override
            public void onMarkerReached(AudioRecord recorder)
            {
                Log.d(TAG, "marker reached");
            }
        };
        recorder.setRecordPositionUpdateListener(positionUpdater);

        Log.d(TAG, "start recording, bufferSize: " + bufferSize);
        recorder.startRecording(); 

//remember to still have a read loop otherwise the listener won't trigger
while (continueRecording)
        {
            recorder.read(audioData, 0, bufferSize);
        }

Beige

+1 nice answer, suppose I want to write the audioData to an AudioTrack object where should I place the call for it.. after the read call inside the listener or after the read call in while loop ?.. plz help!!! – Enthusiasm

Same question : Can you explain the way you handle detectAfterEvery ? – Sapid

I edited the question to discuss detectAfterEvery. Also I provide links to more extensive code – Beige

Isn't the parameter passed to setPositionNotificationPeriod supposed to be in frames, instead of samples? – Ricebird

Answer 4

private int freq =8000;
private AudioRecord audioRecord = null;
private Thread Rthread = null;

private AudioManager audioManager=null;
private AudioTrack audioTrack=null;
byte[] buffer = new byte[freq];

//call this method at start button

protected void Start()

{

loopback();

}

protected void loopback() { 

    android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);
    final int bufferSize = AudioRecord.getMinBufferSize(freq,
            AudioFormat.CHANNEL_CONFIGURATION_MONO,
            AudioFormat.ENCODING_PCM_16BIT);


    audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, freq,
            AudioFormat.CHANNEL_CONFIGURATION_MONO,
            MediaRecorder.AudioEncoder.AMR_NB, bufferSize);

    audioTrack = new AudioTrack(AudioManager.ROUTE_HEADSET, freq,
            AudioFormat.CHANNEL_CONFIGURATION_MONO,
            MediaRecorder.AudioEncoder.AMR_NB, bufferSize,
            AudioTrack.MODE_STREAM);



    audioTrack.setPlaybackRate(freq);
     final byte[] buffer = new byte[bufferSize];
    audioRecord.startRecording();
    Log.i(LOG_TAG, "Audio Recording started");
    audioTrack.play();
    Log.i(LOG_TAG, "Audio Playing started");
    Rthread = new Thread(new Runnable() {
        public void run() {
            while (true) {
                try {
                    audioRecord.read(buffer, 0, bufferSize);                                    
                    audioTrack.write(buffer, 0, buffer.length);

                } catch (Throwable t) {
                    Log.e("Error", "Read write failed");
                    t.printStackTrace();
                }
            }
        }
    });
    Rthread.start();

}

It plays the recorded audio less than 100 ms delay.

Allegiance