onaudioprocess not called on ios11

Asked 22/9, 2017 at 10:53 Answered 2/10, 2017 at 21:36

I am trying to get audio capture from the microphone working on Safari on iOS11 after support was recently added

However, the onaudioprocess callback is never called. Here's an example page:

<html>
    <body>
        <button onclick="doIt()">DoIt</button>
        <ul id="logMessages">
        </ul>
        <script>
            function debug(msg) {
                if (typeof msg !== 'undefined') {
                    var logList = document.getElementById('logMessages');
                    var newLogItem = document.createElement('li');
                    if (typeof  msg === 'function') {
                        msg = Function.prototype.toString(msg);
                    } else if (typeof  msg !== 'string') {
                        msg = JSON.stringify(msg);
                    }
                    var newLogText = document.createTextNode(msg);
                    newLogItem.appendChild(newLogText);
                    logList.appendChild(newLogItem);
                }
            }
            function doIt() {
                var handleSuccess = function (stream) {
                    var context = new AudioContext();
                    var input = context.createMediaStreamSource(stream)
                    var processor = context.createScriptProcessor(1024, 1, 1);

                    input.connect(processor);
                    processor.connect(context.destination);

                    processor.onaudioprocess = function (e) {
                        // Do something with the data, i.e Convert this to WAV
                        debug(e.inputBuffer);
                    };
                };

                navigator.mediaDevices.getUserMedia({audio: true, video: false})
                        .then(handleSuccess);
            }
        </script>
    </body>
</html>

On most platforms, you will see items being added to the messages list as the onaudioprocess callback is called. However, on iOS, this callback is never called.

Is there something else that I should do to try and get it called on iOS 11 with Safari?

Latinist answered 22/9, 2017 at 10:53 Comment(2)

i have same problem =( – Cheka 25/9, 2017 at 7:21

Edit: You must either create the webkitAudioContext() or else call .resume() on it in direct response to a tap, instead of when you get the stream. See my answer below for a fully working version of your code. – Snowdrift 9/10, 2017 at 17:54

There are two problems. The main one is that Safari on iOS 11 seems to automatically suspend new AudioContext's that aren't created in response to a tap. You can resume() them, but only in response to a tap.

(Update: Chrome mobile also does this, and Chrome desktop will have the same limitation starting in version 70 / December 2018.)

So, you have to either create it before you get the MediaStream, or else get the user to tap again later.

The other issue with your code is that AudioContext is prefixed as webkitAudioContext in Safari.

Here's a working version:

<html>
<body>
<button onclick="beginAudioCapture()">Begin Audio Capture</button>
<script>
  function beginAudioCapture() {

    var AudioContext = window.AudioContext || window.webkitAudioContext;    
    var context = new AudioContext();
    var processor = context.createScriptProcessor(1024, 1, 1);
    processor.connect(context.destination);

    var handleSuccess = function (stream) {
      var input = context.createMediaStreamSource(stream);
      input.connect(processor);

      var recievedAudio = false;
      processor.onaudioprocess = function (e) {
        // This will be called multiple times per second.
        // The audio data will be in e.inputBuffer
        if (!recievedAudio) {
          recievedAudio = true;
          console.log('got audio', e);
        }
      };
    };

    navigator.mediaDevices.getUserMedia({audio: true, video: false})
      .then(handleSuccess);
  }
</script>
</body>
</html>

(You can set the onaudioprocess callback sooner, but then you get empty buffers until the user approves of microphone access.)

Oh, and one other iOS bug to watch out for: the Safari on iPod touch (as of iOS 12.1.1) reports that it does not have a microphone (it does). So, getUserMedia will incorrectly reject with an Error: Invalid constraint if you ask for audio there.

FYI: I maintain the microphone-stream package on npm that does this for you and provides the audio in a Node.js-style ReadableStream. It includes this fix, if you or anyone else would prefer to use that over the raw code.

Snowdrift answered 2/10, 2017 at 21:36 Comment(11)

Hi Nathan - I use Promises in my onclick method - I wonder is this confusing Safari that we're no longer in a user tap action? I've tried a few different things in my actual code, and the AudioContext is always in a suspended state – Latinist 3/10, 2017 at 8:47

Yep, a promise resolution is considered a separate event, and it doesn't count as a direct user action, so audio is blocked. Create an audio context in direct response to the tap, and then use it later. – Snowdrift 3/10, 2017 at 12:3

(The whole mess is because Apple doesn't want auto-playing audio on mobile Safari. So they break everything to prevent that.) – Snowdrift 3/10, 2017 at 12:7

Bah! Sorry @nathan-friedly - SO automatically assigned the bounty to a different answer before I had a chance to test this and accept it as the answer. – Latinist 10/10, 2017 at 9:44

even as a response to a user action I got the audioContext in suspended state and had to add audioContext.resume() right after its creation, that's 11.0.3 – Cocotte 13/10, 2017 at 18:10

Hi, thank you for great resource. I have a question. How can we convert raw data to blob which we can replay? Thanks in advance. – Bourgeon 15/10, 2017 at 4:22

@KoOhhashi check out developer.mozilla.org/en-US/docs/Web/API/MediaRecorder and/or github.com/mattdiamond/Recorderjs – Snowdrift 16/10, 2017 at 19:8

@NathanFriedly I didn't instantiate the processor as a direct response to user action, maybe that's maybe why I had to use audioContext.resume – Cocotte 18/10, 2017 at 8:47

@Cocotte yep, that would do it. – Snowdrift 19/10, 2017 at 14:17

a bigger problem for me is that getUserMedia lets you set a sample rate in the audio constraints. However, if the AudioContext is created prior to the getUserMedia the question becomes if this is possible and does it make the sample rate audio constraint a mute point? – Telestich 5/11, 2017 at 17:46

That's a good question, I don't know if it's possible to control the sample rate of the AudioContext, but that would be convenient. Currently, I just take whatever sample rate I get by default (usually 48 or 44.1khz) and downsample it myself - github.com/watson-developer-cloud/speech-javascript-sdk/blob/… – Snowdrift 6/11, 2017 at 18:38

Tried it on iOS 11.0.1, and unfortunately this problem still isn't fixed.

As a workaround, I wonder if it makes sense to replace the ScriptProcessor with a function that takes the steam data from a buffet and then processes it every x milliseconds. But that's a big change to the functionality.

Takeo answered 29/9, 2017 at 12:22 Comment(4)

Yeah I just tried on iOS 11.0.1 myself. I've seen some demos that do what you suggest, but the "buffer" being sent every X milliseconds is always empty (at least in the demos that I've tried). It seems that access to the microphone bytes themselves is the issue. – Latinist 29/9, 2017 at 13:9

What you wrote gave me a few ideas, unfortunately none of them worked so far. I'll still put them here, maybe it helps. You mentioned the buffet being empty, so my thought was the problem might be with getUserMedia. Tried {audio: true, video: true}, and navigator.getUserMedia instead of navigator.mediaDevices.getUserMedia. none of these combinations worked. Also tried to specify the device ID, that also didn't help. – Takeo 29/9, 2017 at 14:9

Unfortunately I can't comment onto Nathan's answer, but his method works. Moving processor = createScriptProcessor and processor.connect before createMediaStremSource does the trick. – Takeo 3/10, 2017 at 11:21

@DanielWu your last comment just fixed my webapp that makes essential use of recording, so I could not be more grateful. Could you maybe edit your answer with it so it would be easier to find for others, as it is glossed over by the others, and somehow seems to make all the difference on iOS Safari. – Grippe 4/10, 2018 at 8:52

Just wondering... do you have the setting enabled in Safari settings? It comes enabled by default in iOS11, but maybe you just disabled it without noticing.

Sewellyn answered 29/9, 2017 at 6:23 Comment(1)

Thanks for the tip @damianmr, but alas camera and microphone access is enabled in Safari, and it will prompt me to allow access the first time I visit the page. it seems everything is working as it should, just the onaudioprocess is never called – Latinist 29/9, 2017 at 9:54

Recommended topics

Hot tags