Correct way to draw/edit a CVPixelBuffer in Swift in iOS

Question

Is there a standard performant way to edit/draw on a CVImageBuffer/CVPixelBuffer in swift?

All the video editing demos I've found online overlay the drawing (rectangles or text) on the screen and don't directly edit the CVPixelBuffer.

UPDATE I tried using a CGContext but the saved video doesn't show the context drawing

private var adapter: AVAssetWriterInputPixelBufferAdaptor?

extension TrainViewController: CameraFeedManagerDelegate {
    
    func didOutput(sampleBuffer: CMSampleBuffer) {

        let time = CMTime(seconds: timestamp - _time, preferredTimescale: CMTimeScale(600))
        let pixelBuffer: CVPixelBuffer? = CMSampleBufferGetImageBuffer(sampleBuffer)

        guard let context = CGContext(data: CVPixelBufferGetBaseAddress(pixelBuffer),
                                  width: width,
                                  height: height,
                                  bitsPerComponent: 8,
                                  bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
                                  space: colorSpace,
                                  bitmapInfo: alphaInfo.rawValue)
        else {
          return nil
        }

        context.setFillColor(red: 1, green: 0, blue: 0, alpha: 1.0)
        context.fillEllipse(in: CGRect(x: 0, y: 0, width: width, height: height))
        context.flush()

        adapter?.append(pixelBuffer, withPresentationTime: time)


    }
}

Jaundice

Answer 1

You need to call CVPixelBufferLockBaseAddress(pixelBuffer, 0) before creating the bitmap CGContext and CVPixelBufferUnlockBaseAddress(pixelBuffer, 0) after you have finished drawing to the context.

Without locking the pixel buffer, CVPixelBufferGetBaseAddress() returns NULL. This causes your CGContext to allocate new memory to draw into, which is subsequently discarded.

Also double check your colour space. It's easy to mix up your components.

e.g.

guard
    CVPixelBufferLockBaseAddress(pixelBuffer) == kCVReturnSuccess,
    let context = CGContext(data: CVPixelBufferGetBaseAddress(pixelBuffer),
                            width: width,
                            height: height,
                            bitsPerComponent: 8,
                            bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
                            space: colorSpace,
                            bitmapInfo: alphaInfo.rawValue)
else {
    return nil
}

context.setFillColor(red: 1, green: 0, blue: 0, alpha: 1.0)
context.fillEllipse(in: CGRect(x: 0, y: 0, width: width, height: height))

CVPixelBufferUnlockBaseAddress(pixelBuffer)

adapter?.append(pixelBuffer, withPresentationTime: time)

Idioplasm

That worked, just needed the buffer lock. – Jaundice

I’m glad! But TBH I’m not sure those are your pixels to modify… oh well YOLO! – Idioplasm

Definitely "fair use" and I created the CVPixelBuffer so ITS MY PRECIOUS lol. Thanks! – Jaundice