AVAssetReaderOutput copyNextSampleBuffer hangs when encoding video w/ audio on device

Question

After implementing the solution to encoding video (with audio) in this question, Video Encoding using AVAssetWriter - CRASHES, I found that the code works correctly in the iPhone Simulator. Unfortunately, certain videos fail to encode their audio while running on an actual iPhone 5 (and other devices).

For example, videos generated from the WWDC 2011 sample code RosyWriter (https://developer.apple.com/library/IOS/samplecode/RosyWriter/Introduction/Intro.html) do not completely encode because the function -[AVAssetReaderOutput copyNextSampleBuffer] never returns.

The video buffers come in correctly, but as soon as it tries to copy the first audio CMSampleBufferRef, the call hangs. When I try this on videos that come from other sources, like those recorded in the native iOS Camera app, the audio imports correctly.

This thread, https://groups.google.com/forum/#!topic/coreaudio-api/F4cqCu99nUI, makes note of the copyNextSampleBuffer function hanging when being used with AudioQueues, and suggests keeping the operations on a single thread. I've tried keeping everything on a separate thread, on the main thread, but had no luck.

Did anyone else experience this and have a possible solution?

EDIT: It appears that videos generated from RosyWriter have their tracks reversed relative to videos from the native Camera app, i.e. audio stream as stream 0, and video stream as stream 1.

Stream #0:0(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 60 kb/s
Metadata:
  creation_time   : 2013-10-28 16:13:05
  handler_name    : Core Media Data Handler
Stream #0:1(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080, 8716 kb/s, 28.99 fps, 29.97 tbr, 600 tbn, 1200 tbc
Metadata:
  rotate          : 90
  creation_time   : 2013-10-28 16:13:05
  handler_name    : Core Media Data Handler

Not sure if this makes a difference to the AVAssetReader.

Mucosa

Did you ever find a solution for this? I'm experiencing the same issue. I have been reverse-engineering the RosyWriter to record my own videos, and when I try to encode/export them later, they 'sometimes' fail. It's very very inconsistent.. – Biconvex

It seemed to be an issue with the way AVFoundation was encoding the audio track. It's fixed in iOS8. I'll add an answer. – Mucosa

Is there any recent development on this issue? – Thicket

Answer 1

This was an AVFoundation bug in iOS7. It is now fixed in iOS8.

Mucosa

It's good news that it's fixed in iOS8, but for those of us who want to support earlier iOS-versions, it's still a problem :( I too have Audio as the first stream, video as the second, in my files. Did you find out if this had any significance to the error? – Biconvex

Not sure if it's related to the error, but the videos that record correctly in iOS8 now have video stream first, and audio stream second. – Mucosa

Did anybody find a workaround for this issue? Is there maybe a way to change the order of the streams when setting up the AssetWriter? I'd really like to be able to re-process a video I recorded before – also in iOS 7. – Pentacle

I tested this on iOS8 on my iPhone 5 and it is NOT WORKING. Works fine on simulator (NOT on the iPhone 5). Perhaps it works on iPhone 6? – Moniliform

I'm seeing copyNextBuffer occasionally hang on iOS 9.3.3 as well. For me, it's occurring sporadically with an AVAssetReaderAudioMixOutput with an asset that has 5 audio tracks, all with non-empty time ranges. Anyone else still seeing this? – Torp

Still a problem on iOS 13, new hardware. – Mellette

Answer 2

I was still experiencing this issue on iOS 9.3.2, and the thing that resolved it was making sure that the AVAssetReaderAudioMixOutput* was initially set with options rather than nil when calling -[AVAssetReaderAudioMixOutput assetReaderAudioMixOutputWithAudioTracks].

Example:

NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:
                                    [NSNumber numberWithInt:kAudioFormatLinearPCM], AVFormatIDKey,
                                    [NSNumber numberWithFloat:44100.0], AVSampleRateKey,
                                    [NSNumber numberWithInt:16], AVLinearPCMBitDepthKey,
                                    [NSNumber numberWithBool:NO], AVLinearPCMIsNonInterleaved,
                                    [NSNumber numberWithBool:NO], AVLinearPCMIsFloatKey,
                                    [NSNumber numberWithBool:NO], AVLinearPCMIsBigEndianKey,
                                    nil];

// create an AVAssetReaderOutput for the audio tracks
NSArray* audioTracks = [asset tracksWithMediaType:AVMediaTypeAudio];
AVAssetReaderAudioMixOutput* _audioReaderOutput = [AVAssetReaderAudioMixOutput assetReaderAudioMixOutputWithAudioTracks:audioTracks audioSettings:outputSettings];

This prevented later calls to -[AVAssetReaderOutput copyNextSampleBuffer] from hanging when they otherwise were doing so.

Textile

Answer 3

Check the time range on the audio tracks using something like this:

NSLog(@"audioTrack timeRange: %lld, %lld", audioTrack.timeRange.start.value, audioTrack.timeRange.duration.value);

Empty time ranges (0, 0) can cause copyNextSampleBuffer() to hang.

Trainband

Thanks for the suggestion, but I have videos with a non-empty time range that still fail. – Mucosa

Did you check each individual track in the video file? – Trainband

Yes. There is only one video track and one audio track in the video file, and they both have non-empty time ranges. – Mucosa

Answer 4

This is a a hardware specific issue. I tested it on the simulator, iPhone5, and iPhone 6. The problem occurs 100% with certain video files, but ONLY on the iPhone 5. I can't speak for the iPhone 5S though. iOS8 does NOT fix this issue. The fix may be having to limit your iPhone 5 users from certain features. Another work around is to have the user select a video from their camera roll, thus forcing Apple's special compression on it - and baptizing that video into a format that the iPhone 5 hardware can work with.

Moniliform

Occurring years later on all sorts of devices. – Mellette