Rendering small CIImage centered in MTKView

Question

I'm rendering a CIImage to MTKView and the image is smaller than the drawable.

let centered = image.transformed(by: CGAffineTransform(translationX: (view.drawableSize.width - image.extent.width) / 2, y: (view.drawableSize.height - image.extent.height) / 2))
context.render(centered, to: drawable.texture, commandBuffer: buffer, bounds: centered.extent, colorSpace: CGColorSpaceCreateDeviceRGB())

I'd expect the code above to render the image in the center of the view, but the image is positioned at origin instead.

Here's the repo illustrating the problem: https://github.com/truemetal/centered-render-of-ciimage-to-mtkview

Before blaming Metal or CoreImage I'd like to make sure I'm not doing something wrong.

I'd appreciate a link to the documentation that says I can't do something like that.

I can workaround this by compositing the image over another one that would be exactly the size of the drawable like so, but I'm still interested in why exactly the code above does not work.

let centered = image.transformed(by: CGAffineTransform(translationX: (view.drawableSize.width - image.extent.width) / 2, y: (view.drawableSize.height - image.extent.height) / 2))
let background = CIImage(color: .white).cropped(to: CGRect(origin: .zero, size: view.drawableSize))
let preparedImage = centered.composited(over: background)
self.context.render(preparedImage, to: drawable.texture, commandBuffer: buffer, bounds: preparedImage.extent, colorSpace: CGColorSpaceCreateDeviceRGB())

Sherry

Answer 1

This is most curious. If you use the "new" CIRenderDestination API instead of context.render(…) it actually works:

let destination = CIRenderDestination(width: Int(view.drawableSize.width),
                                      height: Int(view.drawableSize.height),
                                      pixelFormat: view.colorPixelFormat,
                                      commandBuffer: buffer,
                                      mtlTextureProvider: { () -> MTLTexture in
                                          return drawable.texture
                                      })
try! self.context.startTask(toRender: centered, to: destination)

I don't know why, but context.render(…) doesn't seem to respect the translation of the image or the given bounds. Maybe someone else knows more...

Mansuetude

Brilliant. That actually proves to me that it's something went sour inside core image. Thanks a lot. – Sherry

I find that this somewhat shorter counterpart works equally well let destination = CIRenderDestination(mtlTexture: drawable.texture, commandBuffer: commandBuffer) – Sherry

Yes, the version with the callback has the advantage of interleaving texture fetch and rendering and is therefore more efficient when rendering multiple frames per second. – Mansuetude

Interesting. Do you by any chance have a reference to the docs to back that point? – Sherry

Oh yes, it was mentioned in the "Advances in Core Image" presentation from WWDC 2017: developer.apple.com/videos/play/wwdc2017/510/?time=1393 – Mansuetude

Ah, the getting currentDrawable in the last possible moment recommendation. I've just revisited that video and you're right, that did mention that. The interesting thing for me, though, is that I don't see any improvement by using that technique in my application - perhaps for the reason that I'm issuing only one render task, and the next line is presentDrawable which anyways captures the drawable. Maybe that technique would help when issuing multiple render tasks. Thanks for the linking the video! – Sherry

I assume it's because what you are doing (at least in the demo application) is super simple and doesn't require much processing anyways. For more complex scenarios it's probably beneficial. – Mansuetude

The demo application is something I've put together to demonstrate a problem with a minimal amount of code. I've tried the technique on a project that uses dozens of core image filters which are applied to the input image and produce one output image that is displayed on a metal view. Then I measure average time per second that is spent on CIContext's startTask(toRender:to:) and it rounds up to around 150-300ms per second using both techniques. My guess is that if there's a dozen of calls to startTask, it would help to reduce waiting by asking for that drawable as late as possible. – Sherry

@FrankRupprecht Please look at this one as well #73907599 – Meatus