Deep Copy of Audio CMSampleBuffer

Question

I am trying to create a copy of a CMSampleBuffer as returned by captureOutput in a AVCaptureAudioDataOutputSampleBufferDelegate.

The problem I am having is that my frames coming from delegate method captureOutput:didOutputSampleBuffer:fromConnection: being dropped after I retain them in CFArray for long time.

Obviously, I need to create deep copies of incoming buffers for further processing. I also know that CMSampleBufferCreateCopy only creates shallow copies.

There are few related questions were asked on SO:

• Pulling data from a CMSampleBuffer in order to create a deep copy
• Creating copy of CMSampleBuffer in Swift returns OSStatus -12743 (Invalid Media Format)
• Deep Copy of CMImageBuffer or CVImageBuffer

But none of them helps me to use correctly CMSampleBufferCreate function with 12 parameters:

  CMSampleBufferRef copyBuffer;

  CMBlockBufferRef data = CMSampleBufferGetDataBuffer(sampleBuffer);
  CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer);
  CMItemCount itemCount = CMSampleBufferGetNumSamples(sampleBuffer);

  CMTime duration = CMSampleBufferGetDuration(sampleBuffer);
  CMTime presentationStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
  CMSampleTimingInfo timingInfo;
  timingInfo.duration = duration;
  timingInfo.presentationTimeStamp = presentationStamp;
  timingInfo.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer);


  size_t sampleSize = CMBlockBufferGetDataLength(data);
  CMBlockBufferRef sampleData;

  if (CMBlockBufferCopyDataBytes(data, 0, sampleSize, &sampleData) != kCMBlockBufferNoErr) {
    VLog(@"error during copying sample buffer");
  }

  // Here I tried data and sampleData CMBlockBuffer instance, but no success
  OSStatus status = CMSampleBufferCreate(kCFAllocatorDefault, data, isDataReady, nil, nil, formatDescription, itemCount, 1, &timingInfo, 1, &sampleSize, &copyBuffer);

  if (!self.sampleBufferArray)  {
    self.sampleBufferArray = CFArrayCreateMutable(NULL, 0, &kCFTypeArrayCallBacks);
    //EXC_BAD_ACCESS crash when trying to add sampleBuffer to the array
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  } else  {
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  }

How do you deep copy Audio CMSampleBuffer? Feel free to use any language (swift/objective-c) in your answers.

Chalaza

There have been multiple questions on SO asking the same question, but none of them have a satisfying solution. I had the same problem in the past and couldn’t get it to work. In theory you could copy the underlying pixelbuffer, but it didn’t work when I last tried. Depending on your context I would suggest just writing to a file instead of buffering and then trimming the output video afterwards. At least that’s what I did. Be sure to post the solution here if you find a way to make this work. – Normand

Is it obvious that you need a deep copy? What happens when you use CMSampleBufferCreateCopy? Would CMSampleBufferCopySampleBufferForRange give you a deep copy? Do you really need CMSampleBuffers for further processing? If you're doing your own processing, length + pointer might be more convenient. – Villalobos

@RhythmicFistman yes, it is obvious that I need to deep copy, if I use CMSampleBufferCreateCopy and then retain the copied sample in CFArray more than 1s, didOutputSampleBuffer stops getting called. You can easily reproduce it with this question. I will check the behavior with CMSampleBufferCopySampleBufferForRange and will update you. – Chalaza

Ah, ok, that retaining buffers blocks your delegate callbacks is important information. Do you have a link to a runnable version of the above code? – Villalobos

Answer 1

Here is a working solution I finally implemented. I sent this snippet to Apple Developer Technical support and asked them to check if it is a correct way to copy incoming sample buffer. The basic idea is copy AudioBufferList and then create a CMSampleBuffer and set AudioBufferList to this sample.

AudioBufferList audioBufferList;
CMBlockBufferRef blockBuffer;
//Create an AudioBufferList containing the data from the CMSampleBuffer,
//and a CMBlockBuffer which references the data in that AudioBufferList.
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer);
NSUInteger size = sizeof(audioBufferList);
char buffer[size];

memcpy(buffer, &audioBufferList, size);
//This is the Audio data.
NSData *bufferData = [NSData dataWithBytes:buffer length:size];

const void *copyBufferData = [bufferData bytes];
copyBufferData = (char *)copyBufferData;

CMSampleBufferRef copyBuffer = NULL;
OSStatus status = -1;

/* Format Description */

AudioStreamBasicDescription audioFormat = *CMAudioFormatDescriptionGetStreamBasicDescription((CMAudioFormatDescriptionRef) CMSampleBufferGetFormatDescription(sampleBuffer));

CMFormatDescriptionRef format = NULL;
status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &audioFormat, 0, nil, 0, nil, nil, &format);

CMFormatDescriptionRef formatdes = NULL;
status = CMFormatDescriptionCreate(NULL, kCMMediaType_Audio, 'lpcm', NULL, &formatdes);
if (status != noErr)
{
  NSLog(@"Error in CMAudioFormatDescriptionCreator");
  CFRelease(blockBuffer);
  return;
}

/* Create sample Buffer */
CMItemCount framesCount = CMSampleBufferGetNumSamples(sampleBuffer);
CMSampleTimingInfo timing   = {.duration= CMTimeMake(1, 44100), .presentationTimeStamp= CMSampleBufferGetPresentationTimeStamp(sampleBuffer), .decodeTimeStamp= CMSampleBufferGetDecodeTimeStamp(sampleBuffer)};

status = CMSampleBufferCreate(kCFAllocatorDefault, nil , NO,nil,nil,format, framesCount, 1, &timing, 0, nil, &copyBuffer);

if( status != noErr) {
  NSLog(@"Error in CMSampleBufferCreate");
  CFRelease(blockBuffer);
  return;
}

/* Copy BufferList to Sample Buffer */
AudioBufferList receivedAudioBufferList;
memcpy(&receivedAudioBufferList, copyBufferData, sizeof(receivedAudioBufferList));

//Creates a CMBlockBuffer containing a copy of the data from the
//AudioBufferList.
status = CMSampleBufferSetDataBufferFromAudioBufferList(copyBuffer, kCFAllocatorDefault , kCFAllocatorDefault, 0, &receivedAudioBufferList);
if (status != noErr) {
  NSLog(@"Error in CMSampleBufferSetDataBufferFromAudioBufferList");
  CFRelease(blockBuffer);
  return;
}

Code-Level Support answer:

This code looks ok (though you’ll want to add some additional error checking). I've successfully tested it in an app that implements the AVCaptureAudioDataOutput delegate captureOutput:didOutputSampleBuffer:fromConnection: method to capture and record audio. The captured audio I'm getting when using this deep copy code appears to be the same as what I get when directly using the provided sample buffer (without the deep copy).

Apple Developer Technical Support

Chalaza

This is great, but I'm running into a similar issue with video samples (I need to retain the sample buffers). Do you have any tips on how to implement it for that? – Restoration

Hi, I'm working on a project where i need to deep copy both video and audio. I've been able to deep copy video samplebuffer but cannot find a good solution for audio in Swift. Did you have the chance to convert your code in swift ? With your code, once you get the samplebuffer copy, how do you manage to write it to a file ? Thanks – Cadman

Answer 2

Couldn't find a decent answer doing this in Swift. Here's an extension:

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        guard let formatDesc = CMSampleBufferGetFormatDescription(self),
              let data = try? self.dataBuffer?.dataBytes() else {
                  return nil
              }
        let nFrames = CMSampleBufferGetNumSamples(self)
        let pts = CMSampleBufferGetPresentationTimeStamp(self)
        let dataBuffer = data.withUnsafeBytes { (buffer) -> CMBlockBuffer? in
            var blockBuffer: CMBlockBuffer?
            let length: Int = data.count
            guard CMBlockBufferCreateWithMemoryBlock(
                allocator: kCFAllocatorDefault,
                memoryBlock: nil,
                blockLength: length,
                blockAllocator: nil,
                customBlockSource: nil,
                offsetToData: 0,
                dataLength: length,
                flags: 0,
                blockBufferOut: &blockBuffer) == noErr else {
                    print("Failed to create block")
                    return nil
                }
            guard CMBlockBufferReplaceDataBytes(
                with: buffer.baseAddress!,
                blockBuffer: blockBuffer!,
                offsetIntoDestination: 0,
                dataLength: length) == noErr else {
                    print("Failed to move bytes for block")
                    return nil
                }
            return blockBuffer
        }
        guard let dataBuffer = dataBuffer else {
            return nil
        }
        var newSampleBuffer: CMSampleBuffer?
        CMAudioSampleBufferCreateReadyWithPacketDescriptions(
            allocator: kCFAllocatorDefault,
            dataBuffer: dataBuffer,
            formatDescription: formatDesc,
            sampleCount: nFrames,
            presentationTimeStamp: pts,
            packetDescriptions: nil,
            sampleBufferOut: &newSampleBuffer
        )
        return newSampleBuffer
    }
}

Mooney

I am trying to make this work. I replaced let data = self.data for let data = try? self.dataBuffer?.dataBytes(). Then I check the output of the description in the original and deep copy objects. Everything is the same except the data buffer which contains a block buffer of 4096 data bytes pointing to another block buffer of 4356 bytes. In the case of the deep copy, the reference to the second block buffer is not there. Any hints? – Maurice

Answer 3

The earlier posted answers are great! I've used LLooggaann's answer succesfully in my codebase and refactored it with modern Swift APIs. This gives much shorter code that also throws errors in case something goes wrong and has a lot less pointer & memory management going on.

For all future devs needing this and wanting a more modern compact implementation:

extension CMSampleBuffer {
    struct InvalidAudioSampleBuffer: Swift.Error {}
    
    func deepCopyAudioSampleBuffer() throws -> CMSampleBuffer {
        guard let formatDescription, let dataBuffer else { throw InvalidAudioSampleBuffer() }

        let data = try dataBuffer.dataBytes()
        let dataBufferCopy = try data.withUnsafeBytes { buffer -> CMBlockBuffer in
            let blockBuffer = try CMBlockBuffer(length: data.count)
            try blockBuffer.replaceDataBytes(with: buffer)
            return blockBuffer
        }

        return try CMSampleBuffer(dataBuffer: dataBufferCopy,
                                  formatDescription: formatDescription,
                                  numSamples: numSamples,
                                  presentationTimeStamp: presentationTimeStamp,
                                  packetDescriptions: [])
    }
}

Arnold

Answer 4

LLooggaann's solution is simpler and works well, however, in case anyone is interested, I migrated the original solution to Swift 5.6:

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        var audioBufferList : AudioBufferList = AudioBufferList()
        var blockBuffer : CMBlockBuffer?

        let sizeOfAudioBufferList = MemoryLayout<AudioBufferList>.size
        
        //Create an AudioBufferList containing the data from the CMSampleBuffer.
        CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(self,
                                                                bufferListSizeNeededOut: nil,
                                                                bufferListOut: &audioBufferList,
                                                                bufferListSize: sizeOfAudioBufferList,
                                                                blockBufferAllocator: nil,
                                                                blockBufferMemoryAllocator: nil,
                                                                flags: 0,
                                                                blockBufferOut: &blockBuffer)

        guard audioBufferList.mNumberBuffers == 1 else { return nil }  //TODO: Make this generic for any number of buffers
        
        /* Deep copy the audio buffer */
        let audioBufferDataSize = Int(audioBufferList.mBuffers.mDataByteSize)
        let audioBuffer = audioBufferList.mBuffers
        let audioBufferDataCopyPointer = UnsafeMutableRawPointer.allocate(byteCount: audioBufferDataSize, alignment: 1)
                
        defer {
            audioBufferDataCopyPointer.deallocate()
        }
        
        memcpy(audioBufferDataCopyPointer, audioBufferList.mBuffers.mData, audioBufferDataSize)
        
        let copiedAudioBuffer = AudioBuffer(mNumberChannels: audioBuffer.mNumberChannels,
                                            mDataByteSize: audioBufferList.mBuffers.mDataByteSize,
                                            mData: audioBufferDataCopyPointer)
        
        /* Create a new audio buffer list with the deep copied audio buffer */
        var copiedAudioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: copiedAudioBuffer)

        /* Copy audio format description, to be used in the new sample buffer */
        guard let sampleBufferFormatDescription = CMSampleBufferGetFormatDescription(self) else { return nil }

        /* Create copy of timing for new sample buffer */
        var duration = CMSampleBufferGetDuration(self)
        duration.value /= Int64(numSamples)
        var timing = CMSampleTimingInfo(duration: duration,
                                        presentationTimeStamp: CMSampleBufferGetPresentationTimeStamp(self),
                                        decodeTimeStamp: CMSampleBufferGetDecodeTimeStamp(self))

        /* New sample buffer preparation, using the audio format description, and the timing information. */
        let sampleCount = CMSampleBufferGetNumSamples(self)
        var newSampleBuffer : CMSampleBuffer?

        guard CMSampleBufferCreate(allocator: kCFAllocatorDefault,
                                   dataBuffer: nil,
                                   dataReady: false,
                                   makeDataReadyCallback: nil,
                                   refcon: nil,
                                   formatDescription: sampleBufferFormatDescription,
                                   sampleCount: sampleCount,
                                   sampleTimingEntryCount: 1,
                                   sampleTimingArray: &timing,
                                   sampleSizeEntryCount: 0,
                                   sampleSizeArray: nil,
                                   sampleBufferOut: &newSampleBuffer) == noErr else { return nil }

        //Create a CMBlockBuffer containing a copy of the data from the AudioBufferList, add to new sample buffer.
        let status = CMSampleBufferSetDataBufferFromAudioBufferList(newSampleBuffer!,
                                                                    blockBufferAllocator: kCFAllocatorDefault,
                                                                    blockBufferMemoryAllocator: kCFAllocatorDefault,
                                                                    flags: 0,
                                                                    bufferList: &copiedAudioBufferList)
        
        guard status == noErr else { return nil }

        return newSampleBuffer
    }
}

Maurice