Prevent SpriteKit Metal Renderer from Exceeding Texture Size limit

Question

I have SpriteKit nodes on which I apply Core Image filters using SKEffectNode. The sprites are images added by the user on the scene, and can be of any size. Some of the filters change the size of the rendered sprites. When that size exceeds the limit allowed by Metal, the app crashes with an error like this:

-[MTLTextureDescriptorInternal validateWithDevice:]:1357: failed assertion `Texture Descriptor Validation
MTLTextureDescriptor has width (9050) greater than the maximum allowed size of 8192.
MTLTextureDescriptor has height (9050) greater than the maximum allowed size of 8192.

How can I prevent any image processing I make in real-time from exceeding Metal's limits?

Here's the SpriteKit code:

var mySprite: SKSpriteNode!
var myEffectNode: SKEffectNode!
var sliderInput: Double = 0

override func didMove(to view: SKView) {    
    // Sprite from an image
    // The user picks the image, it could be of any size 
    mySprite = SKSpriteNode(imageNamed: "myImage")
    mySprite.name = "mySprite"
        
    myEffectNode = SKEffectNode()
    myEffectNode.addChild(mySprite)
    
    addChild(myEffectNode)
}

// Called by SwiftUI
// The filter changes dynamically in real-time

func updateEffects() {
    myEffectNode.filter = CIFilter(name: "CIZoomBlur", parameters: [
        "inputAmount": sliderInput
    ])
}

Desired pseudo code:

func carefullyUpdateEffects() {
    // Compute the image processing
    // Get the texture size limit on this device
    // Check if the expected output of the processing exceeds the limit
        // If no, proceed and render
        // If yes, handle case
}

Kernite

I tried nesting the SKEffectNode inside an SKCropNode, with the crop node acting as a size limiter. But the app still crashes. The renderer seems to render the full processed image anyways, even if only the cropped section is displayed. – Kernite

Don't let a user pick an image that's too big? – Canvas

And you can figure out what "too big" is from this table here: developer.apple.com/metal/Metal-Feature-Set-Tables.pdf – Canvas

@Canvas It's a bit more complicated than that. A Core Image filter can increase the rendered size of the sprite, regardless of the sprite's original size. An image of 80*80 could become 10000*10000 after a CIZoomBlur. – Kernite

You own the code. If it's too big, resize it back ¯\_(ツ)_/¯ – Canvas

@Canvas You don't understand the problem. By the time the output image of the filter exceeds Metal's limit, the renderer has already crashed. – Kernite

Answer 1

I have made progress and would like to share it.

The problem

A Core Image filter may return a result that, when rendered, exceeds Metal's size limit per texture.

Example: A CIFilter(name: "CIZoomBlur", parameters: ["inputAmount": 140]) on an image of 1024x1024 produces an image of 17325*17325. When that filter returns its result to SpriteKit, the renderer crashes.

How to get the filter's expected output size before it is sent to the renderer?

A Solution

Subclass CIFilter and do the check inside that custom class. There, we can override the outputImage property, which is of type CIImage. A CIImage is a Core Image object that represents an image but is not rendered until explicitly asked to. Therefore, we can check its extent.size before the output is sent to the renderer.

The custom class below is a working solution that prevents SpriteKit's renderer from crashing if the applied filter exceeds the renderer's limits. It is based on this answer, which was written to chain filters on the same SpriteKit SKEffectNode. X birds with one stone!

import Core Image

class ChainFilters: CIFilter {
    let chainedFilters: [CIFilter]
    let metalSizeLimit: CGFloat = 8000 /// TBD. Get the texture limit of the device's GPU family, and substract some safety margin
    @objc dynamic var inputImage: CIImage?
    
    init(filters: [CIFilter?]) {
        /// The array of filters can contain a nil if the CIFilter inside it is given a wrong name or parameter
        /// `compactMap { $0 }` filter out any `nil` values from the array
        self.chainedFilters = filters.compactMap { $0 }
        super.init()
    }
    
    /// Override `outputImage` to:
    /// - Chain multiple filters
    /// - Check the output result of each filter before it is passed on
    override var outputImage: CIImage? {
        get {
            let imageKey = "inputImage"
            var workingImage = self.inputImage
            for filter in chainedFilters {
                assert(filter.inputKeys.contains(imageKey))
                filter.setValue(workingImage, forKey: imageKey)
                guard let result = filter.outputImage else {
                    assertionFailure("Filter failed: \(filter.name)")
                    return nil
                }
                
                /// Start Metal limit test
                /// We check the `extent` property of the working image, which is a `CIImage`
                /// A CIImage is an object that represents an image but is not rendered until explicitly asked to
                
                if (result.extent.size.width > metalSizeLimit || result.extent.size.height > metalSizeLimit) {
                    print("Input size = \(workingImage?.extent.size ?? CGSize.zero)")
                    print("Output size = \(result.extent.size)")
                    return workingImage
                }
                /// End Metal limit test
                
                workingImage = result
            }
            /// Here the output image is passed on, ultimately to be rendered in SpriteKit or elsewhere
            return workingImage
        }
    }
    
    required init?(coder aDecoder: NSCoder) { fatalError("init(coder:) has not been implemented") }
}

Example usage in SpriteKit:

myEffectNode.filter = ChainFilters(filters: [
    CIFilter(name: "CIZoomBlur", parameters: ["inputAmount": 140]), // will not render if the result exceeds the limit
    CIFilter(name: "CIPixellate", parameters: ["inputScale": 8])
])

To do

Improve the output checking inside the CIFilter subclass. For example, when the limit is exceeded, return a result with the value that did not make the filter exceed the limit, instead of returning the base unfiltered input image.

Kernite