AVCaptureDeviceOutput not calling delegate method captureOutput

Question

I am building an iOS application (my first) that processes video still frames on the fly. To dive into this, I followed an example from the AV* documentation from Apple.

The process involves setting up an input (the camera) and an output. The output works with a delegate, which in this case is the controller itself (it conforms and implements the method needed).

The problem I am having is that the delegate method never gets called. The code below is the implementation of the controller and it has a couple of NSLogs. I can see the "started" message, but the "delegate method called" never shows.

This code is all within a controller that implements the "AVCaptureVideoDataOutputSampleBufferDelegate" protocol.

- (void)viewDidLoad {

    [super viewDidLoad];

    // Initialize AV session    
        AVCaptureSession *session = [AVCaptureSession new];

        if ([[UIDevice currentDevice] userInterfaceIdiom] == UIUserInterfaceIdiomPhone)
            [session setSessionPreset:AVCaptureSessionPreset640x480];
        else
            [session setSessionPreset:AVCaptureSessionPresetPhoto];

    // Initialize back camera input
        AVCaptureDevice *camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

        NSError *error = nil;

        AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:camera error:&error];

        if( [session canAddInput:input] ){
            [session addInput:input];
        }


    // Initialize image output
        AVCaptureVideoDataOutput *output = [AVCaptureVideoDataOutput new];

        NSDictionary *rgbOutputSettings = [NSDictionary dictionaryWithObject:
                                           [NSNumber numberWithInt:kCMPixelFormat_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey];
        [output setVideoSettings:rgbOutputSettings];
        [output setAlwaysDiscardsLateVideoFrames:YES]; // discard if the data output queue is blocked (as we process the still image)


        //[output addObserver:self forKeyPath:@"capturingStillImage" options:NSKeyValueObservingOptionNew context:@"AVCaptureStillImageIsCapturingStillImageContext"];

        videoDataOutputQueue = dispatch_queue_create("VideoDataOutputQueue", DISPATCH_QUEUE_SERIAL);
        [output setSampleBufferDelegate:self queue:videoDataOutputQueue];


        if( [session canAddOutput:output] ){
            [session addOutput:output];
        }

        [[output connectionWithMediaType:AVMediaTypeVideo] setEnabled:YES];


    [session startRunning];

    NSLog(@"started");


}


- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {

        NSLog(@"delegate method called");

        CGImageRef cgImage = [self imageFromSampleBuffer:sampleBuffer];

        self.theImage.image = [UIImage imageWithCGImage: cgImage ];

        CGImageRelease( cgImage );

}

Note: I'm building with iOS 5.0 as a target.

Edit:

I've found a question that, although asking for a solution to a different problem, is doing exactly what my code is supposed to do. I've copied the code from that question verbatim into a blank xcode app, added NSLogs to the captureOutput function and it doesn't get called. Is this a configuration issue? Is there something I'm missing?

Delude

If there is an error starting your session running (likely since you aren't getting any frames) then it will post a AVCaptureSessionRuntimeErrorNotification notification. Listen for it using [[NSNotificationCenter defaultCenter] addObserver:selector:name:object:]; and when your selector is called, get the ` AVCaptureSessionErrorKey` from the user dictionary to see the error. – Calder

Thanks for your input @Inafziger. I suscribed to AVCaptureSessionRuntimeErrorNotification but it doesn't seem to be triggering :| – Delude

By what means is the view controller created? Does 'started' get output? – Irresolution

@Tommy, the view controller is just attached to a storyboard scene. so it's created by "standard" means. "started" does get output. I've also moved the code to a function that gets triggered by a button to see if it had anything to do with it being in viewDidLoad but no change from that. – Delude

I just copied my code verbatim into a sample project called "SquareCam" I downloaded from the Apple docs. I went through the whole implementation and I gutted it to replace in my implementation. The app runs and the delegate NSLog gets called! But not on my project. I looked to make sure im including all the same frameworks etc, and I can't find any code I'm missing... This is getting weird :( – Delude

Answer 1

Your session is a local variable. Its scope is limited to viewDidLoad. Since this is a new project, I assume it's safe to say that you're using ARC. In that case that object won't leak and therefore continue to live as it would have done in the linked question, rather the compiler will ensure the object is deallocated before viewDidLoad exits.

Hence your session isn't running because it no longer exists.

(aside: the self.theImage.image = ... is unsafe since it performs a UIKit action of the main queue; you probably want to dispatch_async that over to dispatch_get_main_queue())

So, sample corrections:

@implementation YourViewController
{
     AVCaptureSession *session;
}

- (void)viewDidLoad {

    [super viewDidLoad];

    // Initialize AV session    
        session = [AVCaptureSession new];

        if ([[UIDevice currentDevice] userInterfaceIdiom] == UIUserInterfaceIdiomPhone)
            [session setSessionPreset:AVCaptureSessionPreset640x480];
        else
         /* ... etc ... */
}


- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {

        NSLog(@"delegate method called");

        CGImageRef cgImage = [self imageFromSampleBuffer:sampleBuffer];

        dispatch_sync(dispatch_get_main_queue(),
        ^{
            self.theImage.image = [UIImage imageWithCGImage: cgImage ];
            CGImageRelease( cgImage );
         });
}

Most people advocate using an underscore at the beginning of instance variable names nowadays but I omitted it for simplicity. You can use Xcode's built in refactor tool to fix that up after you've verified that the diagnosis is correct.

I moved the CGImageRelease inside the block sent to the main queue to ensure its lifetime extends beyond its capture into a UIImage. I'm not immediately able to find any documentation to confirm that CoreFoundation objects have their lifetime automatically extended when captured in a block.

Irresolution

Thank you very much sir, that totally did it :) - It didn't occur to me to test making session an instance variable. Of course, now I go back to look at SquareCam and it certainly is there too. I've learned a lot of other things from this issue, so it wasn't all a waste. Thanks again! – Delude

I tried that solution and it did not work for me. Can you give me any other insight? – Daw

@Daw I've found another reason why didOutputSampleBuffer delegate method may not be called, check out my answer below. https://mcmap.net/q/271444/-avcapturedeviceoutput-not-calling-delegate-method-captureoutput – Pohai

Answer 2

I've found one more reason why didOutputSampleBuffer delegate method may not be called — save to file and get sample buffer output connections are mutually exclusive. In other words, if your session already has AVCaptureMovieFileOutput and then you add AVCaptureVideoDataOutput, only AVCaptureFileOutputRecordingDelegate delegate methods are called.

Just for the reference, I couldn't find anywhere in AV Foundation framework documentation explicit description of this limitation, but Apple support confirmed this a few years ago, as noted in this SO answer.

One way to solve the problem is to remove AVCaptureMovieFileOutput entirely and manually write recorded frames to the file in didOutputSampleBuffer delegate method, alongside your custom buffer data processing. You may find these two SO answers useful.

Pohai

Does this also apply to using both AVCaptureVideoDataOutput and AVCaptureAudioDataOutput? I have managed to successfully capture frames and write to file using the delegate method for video, but the audio never triggers the delegate method so I have videos with no sound. When I use AVCaptureMovieFileOutput sound is working, so I know it is not my inputs that have the problem. – Calv

Seems iOS supports this now: iOS 15 nope, iOS 16 yes, iOS 17 yes – Speight

Answer 3

In my case the problem is there because I call

if ([_session canAddOutput:_videoDataOutput])
        [_session addOutput:_videoDataOutput];

before I call

[_session startRunning];

I'm just start call addOutput: after startRunning

Hope it's help somebody.

Disclaimer

@nbanic Please stop doing updates just to change the word 'cuz' to 'because', without also doing other improvements on the post. You've edited several answers so far that could do with other significant improvements. – Phagocytosis

Why would calling addOutput come after session startRunning? – Mcburney

The ouput should in fact be added before running the session – Hammerfest

>"Why would calling addOutput come after session startRunning?" | There is no reason. The code is wrong. It helped somehow, maybe because of another mistake! – Disclaimer

Answer 4

My captureOutput function was not called either. And the accepted answer did not exactly point at my problem, as my session was already an instance variable.

BUT, my DispatchQueue for my video frames was local. And the dispatchQueue must ALSO be an instance variable. I don't quite understand why this should be necessary. Perhaps the underlying AVCapture code only keeps a weak pointer to it?

The documentation is very confusing on this.

Carloscarlota