Drawing a CIImage is too slow

Question

I'm creating an app that requires real-time application of filters to images. Converting the UIImage to a CIImage, and applying the filters are both extremely fast operations, yet it takes too long to convert the created CIImage back to a CGImageRef and display the image (1/5 of a second, which is actually a lot if editing needs to be real-time).

The image is about 2500 by 2500 pixels big, which is most likely part of the problem

Currently, I'm using

let image: CIImage //CIImage with applied filters
let eagl = EAGLContext(API: EAGLRenderingAPI.OpenGLES2)
let context = CIContext(EAGLContext: eagl, options: [kCIContextWorkingColorSpace : NSNull()])

//this line takes too long for real-time processing
let cg: CGImage = context.createCGImage(image, fromRect: image.extent)

I've looked into using EAGLContext.drawImage()

context.drawImage(image, inRect: destinationRect, fromRect: image.extent)

Yet I can't find any solid documentation on exactly how this is done, or if it would be any faster

Is there any faster way to display a CIImage to the screen (either in a UIImageView, or directly on a CALayer)? I would like to avoid decreasing the image quality too much, because this may be noticeable to the user.

Jar

Answer 1

I ended up using the context.drawImage(image, inRect: destinationRect, fromRect: image.extent) method. Here's the image view class that I created

import Foundation
//GLKit must be linked and imported
import GLKit

class CIImageView: GLKView{
    var image: CIImage?
    var ciContext: CIContext?

    //initialize with the frame, and CIImage to be displayed
    //(or nil, if the image will be set using .setRenderImage)
    init(frame: CGRect, image: CIImage?){
        super.init(frame: frame, context: EAGLContext(API: EAGLRenderingAPI.OpenGLES2))

        self.image = image
        //Set the current context to the EAGLContext created in the super.init call
        EAGLContext.setCurrentContext(self.context)
        //create a CIContext from the EAGLContext
        self.ciContext = CIContext(EAGLContext: self.context)
    }

    //for usage in Storyboards
    required init?(coder aDecoder: NSCoder){
        super.init(coder: aDecoder)

        self.context = EAGLContext(API: EAGLRenderingAPI.OpenGLES2)
        EAGLContext.setCurrentContext(self.context)
        self.ciContext = CIContext(EAGLContext: self.context)
    }

    //set the current image to image
    func setRenderImage(image: CIImage){
        self.image = image

        //tell the processor that the view needs to be redrawn using drawRect()
        self.setNeedsDisplay()
    }

    //called automatically when the view is drawn
    override func drawRect(rect: CGRect){
        //unwrap the current CIImage
        if let image = self.image{
            //multiply the frame by the screen's scale (ratio of points : pixels),
            //because the following .drawImage() call uses pixels, not points
            let scale = UIScreen.mainScreen().scale
            let newFrame = CGRectMake(rect.minX, rect.minY, rect.width * scale, rect.height * scale)

            //draw the image
            self.ciContext?.drawImage(
                image,
                inRect: newFrame,
                fromRect: image.extent
             )
        }
    }   
}

Then, to use it, simply

let myFrame: CGRect //frame in self.view where the image should be displayed
let myImage: CIImage //CIImage with applied filters

let imageView: CIImageView = CIImageView(frame: myFrame, image: myImage)
self.view.addSubview(imageView)

Resizing the UIImage to the screen size before converting it to a CIImage also helps. It speeds things up a lot in the case of high quality images. Just make sure to use the full-size image when actually saving it.

Thats it! Then, to update the image in the view

imageView.setRenderImage(newCIImage)
//note that imageView.image = newCIImage won't work because
//the view won't be redrawn

Jar

you could use didSet on image property: var image: CIImage? { didSet { self.setNeedsDisplay() } } Then whole setRenderImage method becomes obsolete. – Ehrlich

You must set the context in init?(coder:) (like you did it in init(frame)) in order for this to work from storyboard. – Shoop

Is this recommended now that GLKView has been deprecated since iOS 12? – Logography

@MichaelForrest not recommended of course. You need to use Metal instead. – Leasehold

Answer 2

It may be worth considering Metal and displaying with a MTKView.

You'll need a Metal device which can be created with MTLCreateSystemDefaultDevice(). That's used to create a command queue and Core Image context. Both these objects are persistent and quite expensive to instantiate, so ideally should be created once:

lazy var commandQueue: MTLCommandQueue =
{
    return self.device!.newCommandQueue()
}()

lazy var ciContext: CIContext =
{
    return CIContext(MTLDevice: self.device!)
}()

You'll also need a color space:

let colorSpace = CGColorSpaceCreateDeviceRGB()!

When it comes to rendering a CIImage, you'll need to create a short lived command buffer:

let commandBuffer = commandQueue.commandBuffer()

You'll want to render your CIImage (let's call it image) to the currentDrawable?.texture of a MTKView. If that's bound to targetTexture, the rendering syntax is:

    ciContext.render(image,
        toMTLTexture: targetTexture,
        commandBuffer: commandBuffer,
        bounds: image.extent,
        colorSpace: colorSpace)

    commandBuffer.presentDrawable(currentDrawable!)

    commandBuffer.commit()

I have a working version here.

Hope that helps!

Simon

Zig

I found this the most performant solution. Note that in my code, since I'm not subclassing MTKView, I had to call draw() on the MTKView immediately after calling commit() and also had to set the view's device to the one created and its framebufferOnly to false. – Decretive

It's also worth looking at CIRenderDestination and the startTask method on CIContext. – Zig

I have noticed one annoyance with this function that I'm currently working on addressing, and that's if the view resizes, the texture size of the backing drawable always seems to be one step behind the new size and I cannot seem to find a way to force it to update properly. As it stands, your own view negates this issue using stretching, but I'm creating a graphical application that requires high quality output and I want the image to be rendered at exactly the size of the containing view. – Decretive

OK, I have a solution - you might want to put this into your own linked version. Simply rename the renderImage() function to override func draw(), calling super.draw() at the end of it, and in the didSet of image set needsDisplay to true instead of calling the old function. – Decretive

It's extremely important to call draw() after commandBuffer.commit() or else you'll get a bunch of errors in your console and your app will freeze for several seconds before recovering! – Puente

How do you render transparency of a png? i.e. watermark – Accelerando

you're the best Simon – Arlberg

Answer 3

You can use GlkView and render as you said with context.drawImage() :

let glView = GLKView(frame: superview.bounds, context: EAGLContext(API: .OpenGLES2))
let context = CIContext(EAGLContext: glView.context)

After your processing render the image :

glView.bindDrawable()
context.drawImage(image, inRect: destinationRect, fromRect: image.extent)
glView.display()

Cambodia

Answer 4

That is a pretty big image so that's definitely part of it. I'd recommend looking at GPUImage for doing single image filters. You can skip over using CoreImage altogether.

let inputImage:UIImage  = //... some image
let stillImageSource = GPUImagePicture(image: inputImage)
let filter = GPUImageSepiaFilter()
stillImageSource.addTarget(filter)
filter.useNextFrameForImageCapture()
stillImageSource.processImage()

Absorbed