How efficient is using Combine to show live camera feed on SwiftUI view?

Question

This Kodeco tutorial shows how to display an iOS live camera feed in SwiftUI using Combine.

Here are the essential parts doing this (after stripping away non-essential code lines):

class FrameManager : NSObject, ObservableObject
{
    @Published var current: CVPixelBuffer?

    let videoOutputQueue = DispatchQueue(label: "com.raywenderlich.VideoOutputQ",
                                         qos: .userInitiated,
                                         attributes: [],
                                         autoreleaseFrequency: .workItem)
}


extension FrameManager : AVCaptureVideoDataOutputSampleBufferDelegate
{
    func captureOutput(_ output: AVCaptureOutput,
                       didOutput sampleBuffer: CMSampleBuffer,
                       from connection: AVCaptureConnection)
    {
        if let buffer = sampleBuffer.imageBuffer
        {
            DispatchQueue.main.async
            {
                self.current = buffer
            }
        }
    }
}


extension CGImage
{
    static func create(from cvPixelBuffer: CVPixelBuffer?) -> CGImage?
    {
        guard let pixelBuffer = cvPixelBuffer else
        {
            return nil
        }

        var image: CGImage?
        VTCreateCGImageFromCVPixelBuffer(pixelBuffer, 
                                         options: nil,
                                         imageOut: &image)

        return image
    }
}


class ContentViewModel : ObservableObject
{
    @Published var frame: CGImage?

    private let context = CIContext()

    private let frameManager = FrameManager.shared

    init()
    {
        setupSubscriptions()
    }

    func setupSubscriptions()
    {
        frameManager.$current
            .receive(on: RunLoop.main)
            .compactMap
        { buffer in
            guard let image = CGImage.create(from: buffer) else
            {
                return nil
            }

            var ciImage = CIImage(cgImage: image)

            return self.context.createCGImage(ciImage, from: ciImage.extent)
        }
        .assign(to: &$frame)
    }
}


struct ContentView : View
{
    @StateObject private var model = ContentViewModel()

    var body: some View
    {
        ZStack {
            FrameView(image: model.frame)
                .edgesIgnoringSafeArea(.all)
        }
    }
}


struct FrameView : View
{
    var image: CGImage?

    var body: some View
    {
        if let image = image
        {
            GeometryReader
            { geometry in
                Image(image, scale: 1.0, orientation: .up, label: Text("Video feed"))
                    .resizable()
                    .scaledToFill()
                    .frame(width: geometry.size.width,
                           height: geometry.size.height,
                           alignment: .center)
                    .clipped()
            }
        }
    }
}

Although it's working, is converting each CVPixelBuffer to a SwiftUI Image and showing these on screen using Combine/bindings a good/efficient way to display the live camera feed?

And, what would happen if image processing gets too slow to keep up with the AVCaptureVideoDataOutputSampleBufferDelegate feed; will out of date frames be skipped? (The full code has a few CI filters that slows down things quite a lot.)

Cide

I'm not a SwiftUI expert but the "tutorial" you are following seems to me something of a joke or a deliberate tour de force. The way to display what the device's camera is seeing is with AVCaptureVideoPreviewLayer. The fact that you're using SwiftUI doesn't change anything about that. – Mimesis

@Mimesis I compared this SwiftUI vs. AVCaptureVideoPreviewLayer on iPhone 8: CPU usage: ~30% vs. ~5% and Energy Impact low vs. almost zero. Ergo, the SwiftUI code isn't very bad, but indeed AVCaptureVideoPreviewLayer should be used as it's extremely efficient. – Cide

Answer 1

Long story short, it will be efficient based on your use case.

if you want to preview camera without any filter and set your session.sessionPreset to hd1280x720 or lower; it will work for you. be aware if your app is not consuming memory for other services.

Problems:

The only issue with this approach is memory. When you are capturing 4K, the image size is huge, and you will pass and present it in view. By nature of SwiftUI & @Published property wrapper which is a part of Combine framework, it will update the view each time a new image comes.

Now imagine how much space in memory it needed for 3840x2160. If you add an extra filter layer to each frame by using CIImage and applyingFilter it will cause some lags in the preview and consume a lot of memory. Also, for applying filters, we have to use CIContext(), which is expensive to create.

But by using AVCaptureVideoPreviewLayer we will not have these problems even if we want to preview 4k.

These tests captured while the AVAssetWriter was writing Video/Audio

Suggestion:

My suggestion is to create a view based on UIViewRepresentable and inside that use AVCaptureVideoPreviewLayer as a preview layer to the view.

import SwiftUI
import AVKit

class CameraPreviewView: UIView {
    private var captureSession: AVCaptureSession

    init(session: AVCaptureSession) {
        self.captureSession = session
        super.init(frame: .zero)
    }
    
    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    override class var layerClass: AnyClass {
        AVCaptureVideoPreviewLayer.self
    }

    var videoPreviewLayer: AVCaptureVideoPreviewLayer {
        return layer as! AVCaptureVideoPreviewLayer
    }

    override func didMoveToSuperview() {
        super.didMoveToSuperview()

        if nil != self.superview {
            self.videoPreviewLayer.session = self.captureSession
            self.videoPreviewLayer.videoGravity = .resizeAspectFill
            //Setting the videoOrientation if needed
            self.videoPreviewLayer.connection?.videoOrientation = .landscapeRight
        }
    }
}

struct CameraPreviewHolder: UIViewRepresentable {
    var captureSession: AVCaptureSession
    
    func makeUIView(context: UIViewRepresentableContext<CameraPreviewHolder>) -> CameraPreviewView {
        CameraPreviewView(session: captureSession)
    }

    func updateUIView(_ uiView: CameraPreviewView, context: UIViewRepresentableContext<CameraPreviewHolder>) {
    }

    typealias UIViewType = CameraPreviewView
}

Usage:

You can just pass the session that you created it in camera manager class to the CameraPreviewHolder and add it to your SwiftUI View.

Thorson

thanks for the snippet that's what I'm implementing right now. saved a couple of hours I guess :) – Donalddonaldson