How to apply a Vignette CIFilter to a live camera feed in iOS?

Question

While trying to apply a simple vignette filter to the raw camera feed of an iPhone6, with the help of Metal and Core Image, I see a lot of lag between the frames being processed and rendered in an MTKView

The approach which I have followed is (MetalViewController.swift):

1. Get raw camera output using AVCaptureVideoDataOutputSampleBufferDelegate
2. Convert CMSampleBuffer > CVPixelBuffer > CGImage
3. Create an MTLTexture with this CGImage.

Point no. 2 and 3 are inside the method named: fillMTLTextureToStoreTheImageData

4. Apply a CIFilter to the CIImage fetched from the MTLTexture in the MTKViewDelegate

    func draw(in view: MTKView) {

        if let currentDrawable = view.currentDrawable {
            let commandBuffer = self.commandQueue.makeCommandBuffer()

            if let myTexture = self.sourceTexture{

                let inputImage = CIImage(mtlTexture: myTexture, options: nil)

                self.vignetteEffect.setValue(inputImage, forKey: kCIInputImageKey)

                self.coreImageContext.render(self.vignetteEffect.outputImage!, to: currentDrawable.texture, commandBuffer: commandBuffer, bounds: inputImage!.extent, colorSpace: self.colorSpace)

                commandBuffer?.present(currentDrawable)

                commandBuffer?.commit()
            }
        }
    }

The performance is not at all what Apple mentioned in this doc: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_tasks/ci_tasks.html#//apple_ref/doc/uid/TP30001185-CH3-TPXREF101

Am I missing something?

Crease

I know nothing about the camera part, but here is a fast metal based vignette filter: github.com/mattneub/Programming-iOS-Book-Examples/blob/master/… – Bickerstaff

@Bickerstaff I think I have already achieved this without metal only. Applying a CIFiilter to an Image is very fast via Core Image itself only. Its when we try to process every raw frame from a video, things get out of hand. To fix this I used metal, but it did not work as it was supposed to be. – Crease

Well maybe metal is a red herring here. Apple has sample code showing how to apply a CIFilter effect to a video, though whether it can keep up with live video I would tend to doubt. – Bickerstaff

If you see the link in my question, apple clearly say that for live rendering live frames requires metal or opengl. I have followed the same steps. But what i think is apple has provided a very basic implementation ignoring performance and overhead – Crease

Yes, that could be true. I can only find the FunHouse sample code and the WWDC 2013 session 509 video on this topic. – Bickerstaff

I'm pretty sure your slowness is coming from creating a CGImage from the CVPixelBuffer and then the Metal texture from that, rather than going straight from CVPixelBuffer to MTLTexture. You can use CVMetalTextureCache to do that. – Absa

I also thought the same, and followed this: navoshta.com/metal-camera-part-2-metal-texture but no luck. Will be committing my code soon with MTLTexture code, till then could you please check this link to see if I am going in the right direction? – Crease

@nr5 if there is now way without "performance problem and overhead" when you use CIImage. I think it is beacuse CIContext use CPU for converting CIImage to something. I tested almost all ways – Glidewell

Answer 1

Your step 2 is way too slow to support real-time rendering... and it looks like you're missing a couple of steps. For your purpose, you would typically:

Setup:

1. create a pool of CVPixelBuffer - using CVPixelBufferPoolCreate
2. create a pool of metal textures using CVMetalTextureCacheCreate

For each frame:

3. convert CMSampleBuffer > CVPixelBuffer > CIImage
4. Pass that CIImage through your filter pipeline
5. render the output image into a CVPixelBuffer from the pool created in step 1
6. use CVMetalTextureCacheCreateTextureFromImage to create a metal texture with your filtered CVPixelBuffer

If setup correctly, all these steps will make sure your image data stays on the GPU, as opposed to travelling from GPU to CPU and back to GPU for display.

The good news is all this is demoed in the AVCamPhotoFilter sample code from Apple https://developer.apple.com/library/archive/samplecode/AVCamPhotoFilter/Introduction/Intro.html#//apple_ref/doc/uid/TP40017556. In particular see the RosyCIRenderer class and its superclass FilterRenderer.

Summon

Thanks for the direction. One query, how did you identify when you say "all these steps will make sure your image data stays on the GPU, as opposed to traveling from GPU to CPU and back to GPU for display"? For all I know is an MTLDevice represents GPU operations/computations hence I thought sending my CGImage to MTLTexture and getting that MLTexture in the draw delegate would help increase in the performance. – Crease

I am afraid for "render the output image into a CVPixelBuffer" you must use ciContext.render(ciImage, to: pixelBuffer) and this code run in CPU – Glidewell