AVFoundation audio processing using AVPlayer's MTAudioProcessingTap with remote URLs

Question

There is precious little documentation on AVAudioMix and MTAudioProcessingTap, which allow processing to be applied to the audio tracks (PCM access) of media assets in AVFoundation (on iOS). This article and a brief mention in a WWDC 2012 session is all I have found.

I have got the setup described here working for local media files but it doesn't seem to work with remote files (namely HLS streaming URLs). The only indication that this is expected is the note at the end of this Technical Q&A:

AVAudioMix only supports file-based assets.

Does any one know more about this? is there really no way of accessing the audio PCM data when the asset is not file based? Can anyone find actual Apple documentation relating to MTAudioProcessingTap?

Rocambole

Answer 1

I've noticed quite a few people asking about this around the internet, and the general consensus seemed to be that it wasn't possible.

Turns out it is - I was looking into this for a recent personal project and determined that it is indeed possible to make MTAudioProcessingTap work with remote streams. The trick is to KVObserve the status of the AVPlayerItem; when it's ready to play, you can safely retrieve the underlying AVAssetTrack and set an AudioMix on it.

I did a basic writeup with some (mostly working) code here: http://venodesigns.net/2014/01/08/recording-live-audio-streams-on-ios/

If you already managed to handle this, more power to you, but I figured I'd answer this question since it comes up pretty quickly in Google for this stuff.

Fighterbomber

What stream? I noticed .m3u8's have issues based on some comments on my site. Not sure what the deal there is but can't really look into it at the moment. Edit: Ah, you're the chap from my site - sorry for the repeat! – Fighterbomber

@RyanMcGrath the url is broken could you give a mirror link for sample code, it would really really help me, thanks – Thermometer

@Spacedust_ Apologies! My site was experiencing some issues, should be fine now though. – Fighterbomber

doesn't work in iOS 10 at least. [asset tracksWithMediaType:AVMediaTypeAudio][0]; here the tracks array is nil even after the avplayer is ready to play – Imperturbable