Can I use AVAudioEngine to read from a file, process with an audio unit and write to a file, faster than real-time?
Asked Answered
B

1

18

I am working on an iOS app that uses AVAudioEngine for various things, including recording audio to a file, applying effects to that audio using audio units, and playing back the audio with the effect applied. I use a tap to also write the output to a file. When this is done it writes to the file in real time as the audio is playing back.

Is it possible to set up an AVAudioEngine graph that reads from a file, processes the sound with an audio unit, and outputs to a file, but faster than real time (ie., as fast as the hardware can process it)? The use case for this would be to output a few minutes of audio with effects applied, and I certainly wouldn't want to wait for a few minutes for it to be processed.

Edit: here's the code that I'm using to set up the AVAudioEngine's graph, and play a sound file:

AVAudioEngine* engine = [[AVAudioEngine alloc] init];

AVAudioPlayerNode* player = [[AVAudioPlayerNode alloc] init];
[engine attachNode:player];

self.player = player;
self.engine = engine;

if (!self.distortionEffect) {
    self.distortionEffect = [[AVAudioUnitDistortion alloc] init];
    [self.engine attachNode:self.distortionEffect];
    [self.engine connect:self.player to:self.distortionEffect format:[self.distortionEffect outputFormatForBus:0]];
    AVAudioMixerNode* mixer = [self.engine mainMixerNode];
    [self.engine connect:self.distortionEffect to:mixer format:[mixer outputFormatForBus:0]];
}

[self.distortionEffect loadFactoryPreset:AVAudioUnitDistortionPresetDrumsBitBrush];

NSError* error;
if (![self.engine startAndReturnError:&error]) {
    NSLog(@"error: %@", error);
} else {
    NSURL* fileURL = [[NSBundle mainBundle] URLForResource:@"test2" withExtension:@"mp3"];
    AVAudioFile* file = [[AVAudioFile alloc] initForReading:fileURL error:&error];

    if (error) {
        NSLog(@"error: %@", error);
    } else {
        [self.player scheduleFile:file atTime:nil completionHandler:nil];
        [self.player play];
    }
}

The above code plays the sound in the test2.mp3 file, with the AVAudioUnitDistortionPresetDrumsBitBrush distortion preset applied, in real time.

I then modified the above code by adding these lines after [self.player play]:

        [self.engine stop];
        [self renderAudioAndWriteToFile];

I modified the renderAudioAndWriteToFile method that Vladimir provided so that instead of allocating a new AVAudioEngine in the first line, it simply uses self.engine that has already been set up.

However, in renderAudioAndWriteToFile, it's logging "Can not render audio unit" because AudioUnitRender is returning a status of kAudioUnitErr_Uninitialized.

Edit 2: I should mention that I'm perfectly happy to convert the AVAudioEngine code I posted to use the C apis if that would make things easier. However, I would want the code to produce the same output as the AVAudioEngine code (including the use of the factory preset shown above).

Borrowing answered 6/6, 2015 at 4:56 Comment(0)
T
15
  1. Configure your engine and player node.
  2. Call play method for your player node.
  3. Pause your engine.
  4. Get an audio unit from your AVAudioOutputNode (audioEngine.outputNode) with this method.
  5. Render from audio unit with AudioUnitRender in cycle and write audio buffer list to file with Extended Audio File Services.

Example:

Audio engine configuration

- (void)configureAudioEngine {
    self.engine = [[AVAudioEngine alloc] init];
    self.playerNode = [[AVAudioPlayerNode alloc] init];
    [self.engine attachNode:self.playerNode];
    AVAudioUnitDistortion *distortionEffect = [[AVAudioUnitDistortion alloc] init];
    [self.engine attachNode:distortionEffect];
    [self.engine connect:self.playerNode to:distortionEffect format:[distortionEffect outputFormatForBus:0]];
    self.mixer = [self.engine mainMixerNode];
    [self.engine connect:distortionEffect to:self.mixer format:[self.mixer outputFormatForBus:0]];
    [distortionEffect loadFactoryPreset:AVAudioUnitDistortionPresetDrumsBitBrush];
    NSError* error;
    if (![self.engine startAndReturnError:&error])
        NSLog(@"Can't start engine: %@", error);
    else
        [self scheduleFileToPlay];
}

- (void)scheduleFileToPlay {
    NSError* error;
    NSURL *fileURL = [[NSBundle mainBundle] URLForResource:@"filename" withExtension:@"m4a"];
    self.file = [[AVAudioFile alloc] initForReading:fileURL error:&error];
    if (self.file)
        [self.playerNode scheduleFile:self.file atTime:nil completionHandler:nil];
    else
        NSLog(@"Can't read file: %@", error);
}

Rendering methods

- (void)renderAudioAndWriteToFile {
    [self.playerNode play];
    [self.engine pause];
    AVAudioOutputNode *outputNode = self.engine.outputNode;
    AudioStreamBasicDescription const *audioDescription = [outputNode outputFormatForBus:0].streamDescription;
    NSString *path = [self filePath];
    ExtAudioFileRef audioFile = [self createAndSetupExtAudioFileWithASBD:audioDescription andFilePath:path];
    if (!audioFile)
        return;
    AVURLAsset *asset = [AVURLAsset assetWithURL:self.file.url];
    NSTimeInterval duration = CMTimeGetSeconds(asset.duration);
    NSUInteger lengthInFrames = duration * audioDescription->mSampleRate;
    const NSUInteger kBufferLength = 4096;
    AudioBufferList *bufferList = AEAllocateAndInitAudioBufferList(*audioDescription, kBufferLength);
    AudioTimeStamp timeStamp;
    memset (&timeStamp, 0, sizeof(timeStamp));
    timeStamp.mFlags = kAudioTimeStampSampleTimeValid;
    OSStatus status = noErr;
    for (NSUInteger i = kBufferLength; i < lengthInFrames; i += kBufferLength) {
        status = [self renderToBufferList:bufferList writeToFile:audioFile bufferLength:kBufferLength timeStamp:&timeStamp];
        if (status != noErr)
            break;
    }
    if (status == noErr && timeStamp.mSampleTime < lengthInFrames) {
        NSUInteger restBufferLength = (NSUInteger) (lengthInFrames - timeStamp.mSampleTime);
        AudioBufferList *restBufferList = AEAllocateAndInitAudioBufferList(*audioDescription, restBufferLength);
        status = [self renderToBufferList:restBufferList writeToFile:audioFile bufferLength:restBufferLength timeStamp:&timeStamp];
        AEFreeAudioBufferList(restBufferList);
    }
    AEFreeAudioBufferList(bufferList);
    ExtAudioFileDispose(audioFile);
    if (status != noErr)
        NSLog(@"An error has occurred");
    else
        NSLog(@"Finished writing to file at path: %@", path);
}

- (NSString *)filePath {
    NSArray *documentsFolders =
            NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString *fileName = [NSString stringWithFormat:@"%@.m4a", [[NSUUID UUID] UUIDString]];
    NSString *path = [documentsFolders[0] stringByAppendingPathComponent:fileName];
    return path;
}

- (ExtAudioFileRef)createAndSetupExtAudioFileWithASBD:(AudioStreamBasicDescription const *)audioDescription
                                          andFilePath:(NSString *)path {
    AudioStreamBasicDescription destinationFormat;
    memset(&destinationFormat, 0, sizeof(destinationFormat));
    destinationFormat.mChannelsPerFrame = audioDescription->mChannelsPerFrame;
    destinationFormat.mSampleRate = audioDescription->mSampleRate;
    destinationFormat.mFormatID = kAudioFormatMPEG4AAC;
    ExtAudioFileRef audioFile;
    OSStatus status = ExtAudioFileCreateWithURL(
            (__bridge CFURLRef) [NSURL fileURLWithPath:path],
            kAudioFileM4AType,
            &destinationFormat,
            NULL,
            kAudioFileFlags_EraseFile,
            &audioFile
    );
    if (status != noErr) {
        NSLog(@"Can not create ext audio file");
        return nil;
    }
    UInt32 codecManufacturer = kAppleSoftwareAudioCodecManufacturer;
    status = ExtAudioFileSetProperty(
            audioFile, kExtAudioFileProperty_CodecManufacturer, sizeof(UInt32), &codecManufacturer
    );
    status = ExtAudioFileSetProperty(
            audioFile, kExtAudioFileProperty_ClientDataFormat, sizeof(AudioStreamBasicDescription), audioDescription
    );
    status = ExtAudioFileWriteAsync(audioFile, 0, NULL);
    if (status != noErr) {
        NSLog(@"Can not setup ext audio file");
        return nil;
    }
    return audioFile;
}

- (OSStatus)renderToBufferList:(AudioBufferList *)bufferList
                   writeToFile:(ExtAudioFileRef)audioFile
                  bufferLength:(NSUInteger)bufferLength
                     timeStamp:(AudioTimeStamp *)timeStamp {
    [self clearBufferList:bufferList];
    AudioUnit outputUnit = self.engine.outputNode.audioUnit;
    OSStatus status = AudioUnitRender(outputUnit, 0, timeStamp, 0, bufferLength, bufferList);
    if (status != noErr) {
        NSLog(@"Can not render audio unit");
        return status;
    }
    timeStamp->mSampleTime += bufferLength;
    status = ExtAudioFileWrite(audioFile, bufferLength, bufferList);
    if (status != noErr)
        NSLog(@"Can not write audio to file");
    return status;
}

- (void)clearBufferList:(AudioBufferList *)bufferList {
    for (int bufferIndex = 0; bufferIndex < bufferList->mNumberBuffers; bufferIndex++) {
        memset(bufferList->mBuffers[bufferIndex].mData, 0, bufferList->mBuffers[bufferIndex].mDataByteSize);
    }
}

I used some functions from this cool framework:

AudioBufferList *AEAllocateAndInitAudioBufferList(AudioStreamBasicDescription audioFormat, int frameCount) {
    int numberOfBuffers = audioFormat.mFormatFlags & kAudioFormatFlagIsNonInterleaved ? audioFormat.mChannelsPerFrame : 1;
    int channelsPerBuffer = audioFormat.mFormatFlags & kAudioFormatFlagIsNonInterleaved ? 1 : audioFormat.mChannelsPerFrame;
    int bytesPerBuffer = audioFormat.mBytesPerFrame * frameCount;
    AudioBufferList *audio = malloc(sizeof(AudioBufferList) + (numberOfBuffers-1)*sizeof(AudioBuffer));
    if ( !audio ) {
        return NULL;
    }
    audio->mNumberBuffers = numberOfBuffers;
    for ( int i=0; i<numberOfBuffers; i++ ) {
        if ( bytesPerBuffer > 0 ) {
            audio->mBuffers[i].mData = calloc(bytesPerBuffer, 1);
            if ( !audio->mBuffers[i].mData ) {
                for ( int j=0; j<i; j++ ) free(audio->mBuffers[j].mData);
                free(audio);
                return NULL;
            }
        } else {
            audio->mBuffers[i].mData = NULL;
        }
        audio->mBuffers[i].mDataByteSize = bytesPerBuffer;
        audio->mBuffers[i].mNumberChannels = channelsPerBuffer;
    }
    return audio;
}

void AEFreeAudioBufferList(AudioBufferList *bufferList ) {
    for ( int i=0; i<bufferList->mNumberBuffers; i++ ) {
        if ( bufferList->mBuffers[i].mData ) free(bufferList->mBuffers[i].mData);
    }
    free(bufferList);
}
Thurnau answered 6/6, 2015 at 7:55 Comment(20)
Thanks for the advice Vladimir. Unfortunately this code is outputting the message "Can not render audio unit", as the status returned by AudioUnitRender is kAudioUnitErr_Uninitialized. I'll add some code to my question to show how I'm setting up the AVAudioEngine's graph before executing the code you provided.Borrowing
This code just shows you my idea. Unfortunately I can't test it with your configuration. I used a big part of this code in my project exactly for things you are asking. But I worked with Core Audio and AUGraph, not with AVAudioEngine. I sure it will work if you configure your engine right.Thurnau
OK, thanks again Vladimir. I'm happy to switch to using Core Audio and AUGraph, but I'm less familiar with these APIs. I've added a bounty to the question and will happily award it if you can show me how to adapt my existing code to use Core Audio and AUGraph instead of AVAudioEngine.Borrowing
I tried your code and fix an issue. Take a look github.com/VladimirKravchenko/AVAudioEngineOfflineRenderThurnau
This is great - thanks very much Vladimir! Just one problem I've noticed - for at least one of my input files, the resulting output file ends about 2 seconds early. Any idea why that would be? I can post a link to the input file if you like.Borrowing
what is the sample rate of this file?Thurnau
Let us continue this discussion in chat.Borrowing
I dowloaded, built and ran the demo. The resulting file was 40 seconds of silence. I'll try to figure out why, but wanted to give you a heads-up.Sextuplet
mahboudz, what device or simulator did you use? I tested it with my iphone 6 and simulators on 2 different machines and this code worked.Thurnau
I tested on iPhone 6, iOS 8.3. Simulator iPad Air 8.3 iPhone 5: iOS 8.3 iPod Touch: iOS 8.3 Doesn't seem to have a pattern. The silence can be the entire output file, or just the first ~5 seconds, or after ~5 seconds of audio then silence. I can send you the sound files if you want.Sextuplet
Thanks for your comment, I'll try to reproduce and fix this problem. Did you use my sample project github.com/VladimirKravchenko/AVAudioEngineOfflineRender without any changes?Thurnau
Yes. (Sorry for delay. Traveling. I can try to debug once I am home)Sextuplet
I can confirm the same issue @Sextuplet mentioned with the the silence at the beginning of the output file on iPhone 6, iOS8.3 and 8.2 simulators.Curtiscurtiss
I have the same issue with @mahboudz. It's almost 6 second silence in the output file. I tested the project in github with Simulator 9.1Kiger
I've posted a related question about how this approach gives me a silent output if headphones are connected #34144956Between
This is great! Any ideas how one might make this work with a MIDI file using AVAudioSequencer (or MusicPlayer)? I made a quick prototype that successfully produces an audio file when rendered offline, but it sounds as if all the MIDI events are triggered at once: github.com/archagon/aumidisynth-offline-render-testUnguinous
I tried recording an audio (via AVRecorder) and tried loading it, but after rendering, the output file is silent and 0 seconds long. Any clue guys?Hypotension
When my duration is not a whole number for example 15.000090702947846 because I am recording the audio, I get a blank in between. Raised an issue with the cause of having blanks in between at github.com/VladimirKravchenko/AVAudioEngineOfflineRender/issues/…Aubyn
HI . Ive tried this , it doesn't work on iphones 7 , 7+ , it gives me a silenced fileHomovec
How is this different from AVAudioEngine's manual rendering mode?Mandatory

© 2022 - 2024 — McMap. All rights reserved.