How to convert CMSampleBuffer to Data in Swift?

Question

I need to convert CMSampleBuffer to Data format. I am using one Third party framework for audio related task. That framework gives me the streaming (i.e Real Time audio) audio in CMSampleBuffer object.

Like this:

func didAudioStreaming(audioSample: CMSampleBuffer!) {
    //Here I need to conver this to Data format. 
    //Because I am using GRPC framework for Audio Recognization, 
}

Please provide me the steps to convert the CMSampleBuffer to Data.

FYI

    let formatDesc:CMFormatDescription? = CMSampleBufferGetFormatDescription(audioSample)

    <CMAudioFormatDescription 0x17010d890 [0x1b453ebb8]> {
    mediaType:'soun' 
    mediaSubType:'lpcm' 
    mediaSpecific: {
        ASBD: {
            mSampleRate: 16000.000000 
            mFormatID: 'lpcm' 
            mFormatFlags: 0xc 
            mBytesPerPacket: 2 
            mFramesPerPacket: 1 
            mBytesPerFrame: 2 
            mChannelsPerFrame: 1 
            mBitsPerChannel: 16     } 
        cookie: {(null)} 
        ACL: {(null)}
        FormatList Array: {(null)} 
    } 
    extensions: {(null)}
}

Crusade

May i ask how did you record at a sample rate of 16000?? I have called session.setPreferredSampleRate but i keep getting the sample rate in 44100 – Dowel

Answer 1

Try below code to convert CMSampleBuffer to NSData.

let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
CVPixelBufferLockBaseAddress(imageBuffer!, CVPixelBufferLockFlags(rawValue: 0))
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer!)
let height = CVPixelBufferGetHeight(imageBuffer!)
let src_buff = CVPixelBufferGetBaseAddress(imageBuffer!)
let data = NSData(bytes: src_buff, length: bytesPerRow * height)
CVPixelBufferUnlockBaseAddress(imageBuffer!, CVPixelBufferLockFlags(rawValue: 0))

EDIT-

For AudioBuffer use below code -

var audioBufferList = AudioBufferList()
var data = Data()
var blockBuffer : CMBlockBuffer?

CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, nil, &audioBufferList, MemoryLayout<AudioBufferList>.size, nil, nil, 0, &blockBuffer)

let buffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))

for audioBuffer in buffers {
    let frame = audioBuffer.mData?.assumingMemoryBound(to: UInt8.self)
    data.append(frame!, count: Int(audioBuffer.mDataByteSize))
}

Brent

Thanks Nilesh, But I am receiving audio samples not video or image . Will it work for my case? – Crusade

@Crusade I have edited my answer, please let know whether work for you or not. – Brent

Thanks Nileash, I used your updated answer, I am getting - Error Domain=NSOSStatusErrorDomain Code=1954115647 "(null)" this error coming from AudioPlayer – Crusade

let formatDesc:CMFormatDescription? = CMSampleBufferGetFormatDescription(audioSample) <CMAudioFormatDescription 0x17010d890 [0x1b453ebb8]> { mediaType:'soun' mediaSubType:'lpcm' mediaSpecific: { ASBD: { mSampleRate: 48000.000000 mFormatID: 'lpcm' mFormatFlags: 0xc mBytesPerPacket: 4 mFramesPerPacket: 1 mBytesPerFrame: 4 mChannelsPerFrame: 2 mBitsPerChannel: 16 } cookie: {(null)} ACL: {(null)} FormatList Array: {(null)} } extensions: {(null)} – Crusade

The operation couldn’t be completed. (OSStatus error 1954115647.) – Crusade

Where you getting an error, and why are you fetch formatDesc, you should directly pass your data in your audio player like this - let player = try! AVAudioPlayer(data: data) \n player.play() – Brent

I am getting error at playing the let player = try! AVAudioPlayer(data: data) here – Crusade

Okay, Instead of initializing the audio player with the Data object directly, can you please save the file to the Documents folder, and then initialized the player with the file URL. – Brent

Let us continue this discussion in chat. – Crusade

@Crusade is it resolved? I am facing the same issue. – Pabulum

I'm getting a warning for the let buffers = statement. I receive "Initialization of 'UnsafeBufferPointer<AudioBuffer>' results in a dangling buffer pointer" – Quadri

I'm having the same problem. – Expectancy

Answer 2

Using CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer will require to call at some point CFRelease(blockBuffer) because the buffer is retained and if not released the pool of buffers will become eventually empty and no new CMSampleBuffer will be generated.

I'd suggest to get directly the data using the following:

CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t lengthAtOffset;
size_t totalLength;
char *data;
CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &data);

NSData *audioData = [NSData dataWithBytes:data length:totalLength];

Carilyn

This is objective-C, question was asked for Swift :) You could also add swift version – Coraleecoralie

Answer 3

@plamkata__'s code converted to Swift:

var lengthAtOffset = Int()
var totalLength = Int()
var dataPointer: UnsafeMutablePointer<CChar>?

if let blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer) {
    CMBlockBufferGetDataPointer(blockBuffer, atOffset: 0, lengthAtOffsetOut: &lengthAtOffset, totalLengthOut: &totalLength, dataPointerOut: &dataPointer)
    let audioData = Data(bytes: dataPointer!, count: totalLength)
}

Wollongong