Resizing MTKView scales old content before redraw

Question

I'm using a MTKView to draw Metal content. It's configured as follows:

mtkView = MTKView(frame: self.view.frame, device: device)
mtkView.colorPixelFormat = .bgra8Unorm
mtkView.delegate = self
mtkView.sampleCount = 4
mtkView.isPaused = true
mtkView.enableSetNeedsDisplay = true

setFrameSize is overriden to trigger a redisplay.

Whenever the view resizes it scales its old content before it redraws everything. This gives a jittering feeling.

I tried setting the contentGravity property of the MTKView's layer to a non-resizing value, but that totally messes up the scale and position of the content. It seems MTKView doesn't want me to fiddle with that parameter.

How can I make sure that during a resize the content is always properly redrawn?

Chargeable

Does setting layerContentsRedrawPolicy to NSViewLayerContentsRedrawDuringViewResize (.duringViewResize in Swift) help? – Pinpoint

No, I tried a few other options as well, but there is no difference. – Chargeable

How have you configured the MTKView? For example, what are the settings for the paused, enableSetNeedsDisplay, and autoResizeDrawable properties? – Pinpoint

I edited the question to include to include the setup code of the view. – Chargeable

Something else to try: set presentsWithTransaction to true. If that isn't sufficient, you may need to follow the advice in the last paragraph of the docs for that property. The issue is that Metal drawing is asynchronous. You commit a command buffer. It is actually scheduled some time later. If you use its present(_:), then it will call the drawable's present() method at that time. Even that is delayed. It will wait until all rendering to its texture is completed (not just scheduled). – Pinpoint

I tried both the commandBuffer's present as well as the suggestion from the docs with the drawable's present. Both still cause the view the scale the content before the new data is displayed. – Chargeable

With the commandBuffer's present though, the scaled version remains on display until after 5 seconds when the following warning is logged: ----CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces, or set CA_ASSERT_MAIN_THREAD_TRANSACTIONS=1 to abort when an implicit transaction isn't created on a main thread.---- I checked that the draw method is called from the main thread. After this warning, the display is suddenly updated to the correct image. – Chargeable

I think I understand the warning, as the commandBuffer's commit runs on a different thread. – Chargeable

Just to clarify, did you try the combination of setting presentsWithTransaction to true and, in the draw method, after committing the command buffer, doing waitUntilScheduled() and then calling the drawable's present()? Also, can you confirm that your draw method is being called during the resize, hopefully multiple times? – Pinpoint

Yes, I did those commands in that order. To answer your last question, I installed a runloopObserver in setFrameSize and that revealed that the draw method gets called (mostly) after the runloop ends (after the beforeWaiting runloop activity). When I explicitly call the draw method from setFrameSize it actually works! I'm not sue though whether that's a good place to call draw. I'm also afraid that the waitUntilScheduled() call has a performance penalty. Is that true? – Chargeable

Funny enough. This new trick works great for 'normal' view resizing, but when going to fullscreen it won't update the view at all. Not even when both draw() and setNeedsDisplay() are called from setFrameSize(). It will only display the new image data when only setNeedsDisplay() is called.... – Chargeable

Well, I'm not sure what to make of those results. Weird. Yes, the waitUntilScheduled() call will have a performance penalty. You'd only want to do it during the resize. In that case, it would theoretically reduce how quickly the window/view could update in response to the resize to how fast you could draw frames, which is basically a requirement implicit in your question. – Pinpoint

Well, it works for the more general case, which is a big win. Thanks for your support! I'm happy to give you the credits of an accepted answer if you have time to write one. – Chargeable

Answer 1

In my usage of Metal and MTKView, I tried various combinations of presentsWithTransaction and waitUntilScheduled without success. I still experienced occasional frames of stretched content in between frames of properly rendered content during live resize.

Finally, I dropped MTKView altogether and made my own NSView subclass that uses CAMetalLayer and resize looks good now (without any use of presentsWithTransaction or waitUntilScheduled). One key bit is that I needed to set the layer's autoresizingMask to get the displayLayer method to be called every frame during window resize.

Here's the header file:

#import <Cocoa/Cocoa.h>
    
@interface MyMTLView : NSView<CALayerDelegate>    
@end

Here's the implementation:

#import <QuartzCore/CAMetalLayer.h>
#import <Metal/Metal.h>

@implementation MyMTLView

- (id)initWithFrame:(NSRect)frame
{
    if (!(self = [super initWithFrame:frame])) {
        return self;
    }

    // We want to be backed by a CAMetalLayer.
    self.wantsLayer = YES;

    // We want to redraw the layer during live window resize.
    self.layerContentsRedrawPolicy = NSViewLayerContentsRedrawDuringViewResize;

    // Not strictly necessary, but in case something goes wrong with live window
    // resize, this layer placement makes it more obvious what's going wrong.
    self.layerContentsPlacement = NSViewLayerContentsPlacementTopLeft;

    return self;
}

- (CALayer*)makeBackingLayer
{
    CAMetalLayer* metalLayer = [CAMetalLayer layer];
    metalLayer.device = MTLCreateSystemDefaultDevice();
    metalLayer.delegate = self;

    // *Both* of these properties are crucial to getting displayLayer to be
    // called during live window resize.
    metalLayer.autoresizingMask = kCALayerHeightSizable | kCALayerWidthSizable;
    metalLayer.needsDisplayOnBoundsChange = YES;

    return metalLayer;
}

- (CAMetalLayer*)metalLayer
{
    return (CAMetalLayer*)self.layer;
}

- (void)setFrameSize:(NSSize)newSize
{
    [super setFrameSize:newSize];

    self.metalLayer.drawableSize = newSize;
}

- (void)displayLayer:(CALayer*)layer
{
    // Do drawing with Metal.
}

@end

For reference, I do all my Metal drawing in MTKView's drawRect method.

Lyonnais

Unfortunately this didn't work/improve for me. I use the delegate to draw, maybe that's related. – Chargeable

@RemcoPoelstra Hey, I changed my answer completely. I saw some hangs in setFrameSize with my previous answer, so I tried something completely different. – Lyonnais

This answer reduces the frequency of glitches for me but doesn't eliminate them. However I figured out that combining this with presentsWithTransaction and waitUntilScheduled works perfectly. I wrote a blog post and posted a working code sample: thume.ca/2019/06/19/glitchless-metal-window-resizing – Ora

I've been having position and scaling issues that I just couldn't seem to resolve, turns out setting the layerContentsPlacement to TopLeft was exactly the last piece of the puzzle for me. Thanks for providing a half decent example to compare against. – Schreibe

Answer 2

I have the same problem with glitches on view resizing. You can even reproduce it in the HelloTriangle example from the Apple's developer site. However the effect is minimized because the triangle is drawn near the middle of the screen, and it's the content closest to the edge of the window, opposite the corner that drags, that is effected worst. The developer notes regarding use of presentsWithTransaction and waitUntilScheduled do not work for me either.

My solution was to add a Metal layer beneath the window.contentView.layer, and to make that layer large enough that it rarely needs to be resized. The reason this works is that, unlike the window.contentView.layer, which sizes itself automatically to the view (in turn maintaining the window size), you have explicit control of the sublayer size. This eliminates the flickering.

Berget

I've been playing around with your suggestion, but I can't make this to work. How do you momentarily prevent the MTKView from resizing? Or how do you update it to the correct size once the parent has redrawn? – Chargeable

Actually, the solution I have settled on is simpler than the one I suggested here. I just choose a generous layer size, extending beyond the typical size of the view. When the view is resized, it just reveals more of the layer, but never tries to resize it. You'll need your metal layer to be a subLayer of the view's layer. – Berget

Does that mean that you use a normal view instead of a MTKView? Is the performance acceptable while you draw ‘too much’? – Chargeable

I don't bother with the MTKView class, just create a layer and draw into when things change. I don't know what you mean by draw too much. – Berget

Sure, no problem. – Berget

Answer 3

This helped me - https://github.com/trishume/MetalTest

He uses MetalLayer and careful setting of various properties. Everything is pretty smooth even with two side by side in synchronised scroll views with 45megapixel images.

A link to my original problem How do I position an image correctly in MTKView?

Orchidaceous