Applying MPSImageGaussianBlur with depth data

Question

I am trying to create an imitation of the portrait mode in Apple's native camera.

The problem is, that applying the blur effect using CIImage with respect to depth data, is too slow for the live preview I want to show to the user.

My code for this is mission is:

func blur(image: CIImage, mask: CIImage, orientation: UIImageOrientation = .up, blurRadius: CGFloat) -> UIImage? {
    let start = Date()
    let invertedMask = mask.applyingFilter("CIColorInvert")
    
    let output = image.applyingFilter("CIMaskedVariableBlur", withInputParameters: ["inputMask" : invertedMask,
                                                                                    "inputRadius": blurRadius])
    
    guard let cgImage = context.createCGImage(output, from: image.extent) else {
        return nil
    }
    let end = Date()
    let elapsed = end.timeIntervalSince1970 - start.timeIntervalSince1970
    print("took \(elapsed) seconds to apply blur")
    return UIImage(cgImage: cgImage, scale: 1.0, orientation: orientation)
}

I want to apply the blur on the GPU for better performance. For this task, I found this implementation provided by Apple here.

So in Apple's implementation, we have this code snippet:

/** Applies a Gaussian blur with a sigma value of 0.5.
 This is a pre-packaged convolution filter.
 */
class GaussianBlur: CommandBufferEncodable {
    let gaussian: MPSImageGaussianBlur

    required init(device: MTLDevice) {
        gaussian = MPSImageGaussianBlur(device: device,
                                    sigma: 5.0)
    }

    func encode(to commandBuffer: MTLCommandBuffer, sourceTexture: MTLTexture, destinationTexture: MTLTexture) {
        gaussian.encode(commandBuffer: commandBuffer,
                    sourceTexture: sourceTexture,
                    destinationTexture: destinationTexture)
    }
}

How can I apply the depth data into the filtering through the Metal blur version? Or in other words - how can I achieve the first code snippets functionality, with the performance speed of the second code snippet?

Threequarter

Did you find a good way to do this with MPS? As far as I can tell, this could be done by applying the CIFilter chain to a CIImage and rendering it to a MTKView with a metal-backed CIContext, or just writing custom shaders. I'm working on something similar with depth filters, so would love an update. – Dissociate

Hey Michael, sorry but as of now I don’t have any update regarding this – Threequarter

Ah well, thanks anyway. FYI for the time being, I have had pretty good success rendering a filtered CIImage to an MTKView, with pretty much real-time performance.. – Dissociate

Good to hear! Learning some MTK myself right now. I guess there is no escape learning Metal deeply in order to achieve tasks like in this question – Threequarter

@Dissociate 5 years later.. did you continue to use "CIMaskedVariableBlur" rendering a CIImage to an MTKView? I have it working but the FPS slowly tanks over the first 30 secs! – Sausa

Answer 1

For anyone still looking you need to get currentDrawable first in draw(in view: MTKView) method. Implement MTKViewDelegate

func makeBlur() {
    device = MTLCreateSystemDefaultDevice()
    commandQueue = device.makeCommandQueue()
    
    selfView.mtkView.device = device
    selfView.mtkView.framebufferOnly = false
    selfView.mtkView.delegate = self
    
    let textureLoader = MTKTextureLoader(device: device)
    if let image = self.backgroundSnapshotImage?.cgImage, let texture = try? textureLoader.newTexture(cgImage: image, options: nil) {
        sourceTexture = texture
    }
}

func draw(in view: MTKView) {
    if let currentDrawable = view.currentDrawable,
       let commandBuffer = commandQueue.makeCommandBuffer() {
           let gaussian = MPSImageGaussianBlur(device: device, sigma: 5)
           gaussian.encode(commandBuffer: commandBuffer, sourceTexture: sourceTexture, destinationTexture: currentDrawable.texture)
           commandBuffer.present(currentDrawable)
           commandBuffer.commit()
    }
}

Leu