Set rate at which AVSampleBufferDisplayLayer renders sample buffers

Question

I am using an AVSampleBufferDisplayLayer to display CMSampleBuffers which are coming over a network connection in the h.264 format. Video playback is smooth and working correctly, however I cannot seem to control the frame rate. Specifically, if I enqueue 60 frames per second in the AVSampleBufferDisplayLayer it displays those 60 frames, even though the video is being recorded at 30 FPS.

When creating sample buffers, it is possible to set the presentation time stamp by passing a timing info array to CMSampleBufferCreate (the timing info is not present in the h.264 stream but can be calculated or passed in a container format). The presentation time stamps I set are about 0.033 seconds apart and the duration is 0.033 but the display layer still displays as many frames per second as it can.

There are two ways to enqueue buffers on AVSampleBufferDisplayLayer: "constrained" by calling -[AVSampleBufferDisplayLayer enqueueSampleBuffer:] whenever a buffer is ready, or "unconstrained" by calling -[AVSampleBufferDisplayLayer requestMediaDataWhenReadyOnQueue:usingBlock:] and enqueuing the buffers in that block. I've tried both but even the second method displays buffers as quickly as it can - for instance if I have 300 frames queued up on the receiving side then the first time the block in the method above is executed readyForMoreMediaData remains true no matter how many buffers get enqueued, and they are all displayed in a very short time.

This behavior is similar to what one would expect if the kCMSampleAttachmentKey_DisplayImmediately attachment were set on the CMSampleBuffer, however this is NOT currently set (and the default is false).

I tried setting the layers controlTimeBase, but it didn't seem to have any effect. I'm at a loss of other things to try and could not find examples online. Does anyone know how one may control the framerate at which AVSampleBufferDisplayLayer displays frames?

Precipitous

Did you ever find a solution for this? – Prehensile

I'm sorry to say I didn't – Precipitous

how bout now in 2020 :) ? – Norine

Ahalan @OzShabat. I never did. I'm afraid this project is long since dead, and due to other dependencies changing won't even compile anymore, so I'm unable to test proposed solutions. – Precipitous

Answer 1

The Timebase needs to be set to the presentation time stamp (pts) of the first frame you intend to decode. I was indexing the pts of the first frame to 0 by subtracting the initial pts from all subsequent pts and setting the Timebase to 0. For whatever reason, that didn't work.

You want something like this (called before a call to decode):

CMTimebaseRef controlTimebase;
CMTimebaseCreateWithMasterClock( CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &controlTimebase );

displayLayer.controlTimebase = controlTimebase;

// Set the timebase to the initial pts here
CMTimebaseSetTime(displayLayer.controlTimebase, CMTimeMake(ptsInitial, 1));
CMTimebaseSetRate(displayLayer.controlTimebase, 1.0);

Set the PTS for the CMSampleBuffer...

CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

And maybe make sure display immediately isn't set....

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanFalse);

This is covered very briefly in WWDC 2014 Session 513.

Prehensile

Answer 2

Set output presentation timestamp on sample buffer before enqueuing it to AVSampleBufferDisplayLayer. For FPS = 30:

CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer,    CMTimeAdd(kCMTimeZero, CMTimeMake(1 * _frameCount, 30)));

Set the timestamp on first buffer to 0 (kCMTimeZero) and increment the timestamp on subsequent buffers by 1/30th of a second.

Also, need to set timebase on AVSampleBufferDisplayLayer instance so that buffer with presentation timestamp set to 0 is displayed first.

CMTimebaseRef controlTimebase;
CMTimebaseCreateWithMasterClock(CFAllocatorGetDefault(),    CMClockGetHostTimeClock(), &controlTimebase);

_videoLayer.controlTimebase = controlTimebase;
CMTimebaseSetTime(_videoLayer.controlTimebase, kCMTimeZero);
CMTimebaseSetRate(_videoLayer.controlTimebase, 1.0);

Aluminous

Answer 3

Hit with the same issue, managed to play several streams one after another without lags with following timing in CMSampleBufferCreate creation

CMSampleTimingInfo timingdata ={
 .presentationTimeStamp = CMTimeMakeWithSeconds(self.frame0time+(1/self.frameFPS)*self.frameActive, 1000),
 .duration =  CMTimeMakeWithSeconds(1/self.frameFPS, 1000),
 .decodeTimeStamp = kCMTimeInvalid
};

No need to use kCMSampleAttachmentKey_DisplayImmediately with this approach, you just have to self.frameActive++ on every Iframe and BFrame and make self.frame0time = CACurrentMediaTime(); on first frame

Doublehung

How do I know first0time? – Voter