precise timing with AVMutableComposition

Question

I'm trying to use AVMutableComposition to play a sequence of sound files at precise times.

When the view loads, I create the composition with the intent of playing 4 sounds evenly spaced over 1 second. It shouldn't matter how long or short the sounds are, I just want to fire them at exactly 0, 0.25, 0.5 and 0.75 seconds:

AVMutableComposition *composition = [[AVMutableComposition alloc] init];
NSDictionary *options = @{AVURLAssetPreferPreciseDurationAndTimingKey : @YES};

for (NSInteger i = 0; i < 4; i++)
{
  AVMutableCompositionTrack* track = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
  NSURL *url = [[NSBundle mainBundle] URLForResource:[NSString stringWithFormat:@"sound_file_%i", i] withExtension:@"caf"];
  AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url options:options];
  AVAssetTrack *assetTrack = [asset tracksWithMediaType:AVMediaTypeAudio].firstObject;
  CMTimeRange timeRange = [assetTrack timeRange];
  Float64 t = i * 0.25;
  NSError *error;
  BOOL success = [track insertTimeRange:timeRange ofTrack:assetTrack atTime:CMTimeMakeWithSeconds(t, 1) error:&error];
  if (!success)
  {
    NSLog(@"unsuccesful creation of composition");
  }
  if (error)
  {
    NSLog(@"composition creation error: %@", error);
  }
}

AVPlayerItem* playerItem = [AVPlayerItem playerItemWithAsset:composition];
self.avPlayer = [[AVPlayer alloc] initWithPlayerItem:playerItem];

The composition is created successfully with no errors. Later, when I want to play the sequence I do this:

[self.avPlayer seekToTime:CMTimeMakeWithSeconds(0, 1)];
[self.avPlayer play];

For some reason, the sounds are not evenly spaced at all - but play almost all at once. I tried the same thing spaced over 4 seconds, replacing the time calculation like this:

Float64 t = i * 1.0;

And this plays perfectly. Any time interval under 1 second seems to generate unexpected results. What am I missing? Are AVCompositions not supposed to be used for time intervals under 1 second? Or perhaps I'm misunderstanding the time intervals?

Caber

Answer 1

Your CMTimeMakeWithSeconds(t, 1) is in whole second 'slices' because your timescale is set to 1. No matter what fraction t is, the atTime: will always end up as 0. This is why it works when you increase it to 1 second (t=i*1).

You need to set the timescale to 4 to get your desired 0.25 second slices. Since the CMTime is now in .25 second slices, you won't need the i * 0.25 calculcation. Just use the i directly; atTime:CMTimeMake(i, 4)

If you might need to get more precise in the future, you should account for it now so you won't have to adjust your code later. Apple recommends using a timescale of 600 as it is a multiple of the common video framerates (24, 25, and 30 FPS) but it works fine for audio-only too. So for your situation, you would use 24 slices to get your .25 second value; Float64 t = i * 24; atTime:CMTimeMake(t, 600)

As for your issue of all 4 sounds playing almost all at once, be aware of this unanswered SO question where it only happens on the first play. Even with the changes above, you might still run into this issue.

Sirdar

Thanks, this is what I was missing. One correction here: CMTimeMakeWithSeconds should just be CMTimeMake. This helped me understand the difference. – Caber

Answer 2

Unless each track is exactly 0.25 seconds long this is your problem:

Float64 t = i * 0.25;
NSError *error;
BOOL success = [track insertTimeRange:timeRange ofTrack:assetTrack atTime:CMTimeMakeWithSeconds(t, 1) error:&error];

You need to be keeping track of the cumulative time range added so far, and inserting the next track at that time:

CMTime currentTime = kCMTimeZero;

for (NSInteger i = 0; i < 4; i++) {

   /* Code to create track for insertion */

    CMTimeRange trackTimeRange = [assetTrack timeRange];

    BOOL success = [track insertTimeRange:trackTimeRange
                                  ofTrack:assetTrack 
                                    atTime:currentTime
                                      error:&error];

    /* Error checking code */

    //Update time range for insertion
    currentTime = CMTimeAdd(currentTime,trackTimeRange.duration);
}

Ukrainian

Thanks for the answer. One correction in your code, trackTimeRange needs to be trackTimeRange.duration in the calculation for currentTimeRange (needs to be a CCTime). However this actually isn't what I'm trying to do: I want to play the samples at exactly 0, 0.25, 0.5 and 0.75 seconds regardless of their duration. They may overlap, or they may have space between them. Your solution plays the sounds in sequence but waits for each to be finished before firing the next. – Caber

Ah yes sorry...I also renamed currentTimeRange to currentTime as its actually a CMTime object of course. Based on your updated question, the answer is rather more complicated. I'll post an updated solution. – Ukrainian

Answer 3

i changed a bit your code, sorry i had no time to test it.

   AVMutableComposition *composition = [AVMutableComposition composition];
    NSDictionary *options = @{AVURLAssetPreferPreciseDurationAndTimingKey : @YES};

    CMTime totalDuration = kCMTimeZero;

    for (NSInteger i = 0; i < 4; i++)
    {
        AVMutableCompositionTrack* track = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];


        NSURL *url = [NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:[NSString stringWithFormat:@"Record_%i", i] ofType:@"caf"]];
        AVURLAsset *asset = [[AVURLAsset alloc] initWithURL:url options:options];
        AVAssetTrack *assetTrack = [[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
        CMTimeRange timeRange = [assetTrack timeRange];
        NSError *error;

        BOOL success = [track insertTimeRange:timeRange ofTrack:assetTrack atTime:CMTIME_COMPARE_INLINE(totalDuration, >, kCMTimeZero)? CMTimeAdd(totalDuration, CMTimeMake(1, 4)): totalDuration error:&error];

        if (!success)
        {
            NSLog(@"unsuccesful creation of composition");
        }
        if (error)
        {
            NSLog(@"composition creation error: %@", error);
        }
        totalDuration = CMTimeAdd(CMTimeAdd(totalDuration,CMTimeMake(1, 4)), asset.duration);
    }

    AVPlayerItem* playerItem = [AVPlayerItem playerItemWithAsset:composition];
    self.avPlayer = [[AVPlayer alloc] initWithPlayerItem:playerItem];

P.S. use kCMTimeZero instead of CMTimeMakeWithSeconds(0, 1).

Henka

This doesn't successfully create a composition. AVFoundationErrorDomain Code=-11800 – Caber

And the time calculations are wrong here. For index 1 through 3, the CMTime is (value = 0, timescale = 0, flags = 0, epoch = 0), which isn't right. – Caber