Metal core image kernel with sampler

Question

I am trying to use a CIColorKernel or CIBlendKernel with sampler arguments but the program crashes. Here is my shader code which compiles successfully.

extern "C" float4 wipeLinear(coreimage::sampler t1, coreimage::sampler t2, float time) {
    float2 coord1 = t1.coord();
    float2 coord2 = t2.coord();

    float4 innerRect = t2.extent();

    float minX = innerRect.x + time*innerRect.z;
    float minY = innerRect.y + time*innerRect.w;
    float cropWidth = (1 - time) * innerRect.w;
    float cropHeight = (1 - time) * innerRect.z;

    float4 s1 = t1.sample(coord1);
    float4 s2 = t2.sample(coord2);

   if ( coord1.x > minX && coord1.x < minX + cropWidth && coord1.y > minY && coord1.y <= minY + cropHeight) {
       return s1;
   } else {
      return s2;
   }
}

And it crashes on initialization.

class CIWipeRenderer: CIFilter {
var backgroundImage:CIImage?
var foregroundImage:CIImage?
var  inputTime: Float = 0.0

static var kernel:CIColorKernel = { () -> CIColorKernel in

    let url = Bundle.main.url(forResource: "AppCIKernels", withExtension: "ci.metallib")!
    let data = try! Data(contentsOf: url)
    return try! CIColorKernel(functionName: "wipeLinear", fromMetalLibraryData: data) //Crashes here!!!!
    
}()

override var outputImage: CIImage? {
    guard let backgroundImage = backgroundImage else {
        return nil
    }
    
    guard let foregroundImage = foregroundImage else {
        return nil
    }
    
    return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, arguments: [backgroundImage, foregroundImage, inputTime])
}

}

It crashes in the try line with the following error:

 Fatal error: 'try!' expression unexpectedly raised an error: Foundation._GenericObjCError.nilError

If I replace the kernel code with the following, it works like a charm:

  extern "C" float4 wipeLinear(coreimage::sample_t s1, coreimage::sample_t s2, float time)
{
     return mix(s1, s2, time);
}

So there are no obvious errors in the code, such as passing incorrect function name or so.

Chemisorption

Answer 1

Yes, you can't use samplers in CIColorKernel or CIBlendKernel. Those kernels are optimized for the use case where you have a 1:1 mapping from input pixel to output pixel. This allows Core Image to execute multiple of these kernels in one command buffer since they don't require any intermediate buffer writes.
A sampler would allow you to sample the input at arbitrary coordinates, which is not allowed in this case.

You can simply use a CIKernel instead. It's meant to be used when you need to sample the input more freely.

To initialize the kernel, you need to adapt the code like this:

static var kernel: CIKernel = {
    let url = Bundle.main.url(forResource: "AppCIKernels", withExtension: "ci.metallib")!
let data = try! Data(contentsOf: URL)
    return try! CIKernel(functionName: "wipeLinear", fromMetalLibraryData: data)
}()

When calling the kernel, you now need to also provide a ROI callback, like this:

let roiCallback: CIKernelROICallback = { index, rect -> CGRect in
    return rect // you need the same region from the input as the output

}
// or even shorter
let roiCallback: CIKernelROICallback = { $1 }
return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, roiCallback: roiCallback, arguments: [backgroundImage, foregroundImage, inputTime])

Woosley

Please update the answer with code sample how to use CIKernel in this case. – Chemisorption

I started using CIKernel. Something is not correct in my kernel code. It does not produce the correct output. I suspect it is due to coordinate system perhaps or may be I need to pass special samplers as arguments, not sure what is wrong. – Chemisorption

Updated. I also just saw that you can actually use a color kernel for your use case. I'll post another answer shortly. – Woosley

I was able to figure out the issue, it works! But yes I would like to see CIColorKernel as well. – Chemisorption

Nice! See below. – Woosley

Answer 2

For your use case, you actually can use a CIColorKernel. You just have to pass the extent of your render destination to the kernel as well, then you don't need the sampler to access it.

The kernel would look like this:

extern "C" float4 wipeLinear(coreimage::sample_t t1, coreimage::sample_t t2, float4 destinationExtent, float time, coreimage::destination destination) {
    float minX = destinationExtent.x + time * destinationExtent.z;
    float minY = destinationExtent.y + time * destinationExtent.w;
    float cropWidth = (1.0 - time) * destinationExtent.w;
    float cropHeight = (1.0 - time) * destinationExtent.z;

    float2 destCoord = destination.coord();

   if ( destCoord.x > minX && destCoord.x < minX + cropWidth && destCoord.y > minY && destCoord.y <= minY + cropHeight) {
       return t1;
   } else {
      return t2;
   }
}

And you call it like this:

let destinationExtent = CIVector(cgRect: backgroundImage.extent)
return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, arguments: [backgroundImage, foregroundImage, destinationExtent, inputTime])

Note that the last destination parameter in the kernel is passed automatically by Core Image. You don't need to pass it with the arguments.

Woosley

One doubt about extent, what if CIExtent is infinite(which is possible with CIColor generator filter for instance)? – Chemisorption

Yes, that's an issue. Then you need to pick some reference extent. For instance, if you know that your foreground image is always finite, then use that instead. With two infinite images, you wouldn't know where to put the transition line anyway, right? – Woosley

Yes, infinite images are an issue with such transitions. Particularly as in my editing program, images may be coming from a pipeline of operations. I guess cropping infinite extent images to some reference finite extent would be required. – Chemisorption

Answer 3

Bonus answer:

For this blending effect, you actually don't need any kernel at all. You can achieve all that with simple cropping and compositing:

class CIWipeRenderer: CIFilter {
    var backgroundImage:CIImage?
    var foregroundImage:CIImage?
    var inputTime: CGFloat = 0.0

    override var outputImage: CIImage? {
        guard let backgroundImage = backgroundImage else { return nil }
        guard let foregroundImage = foregroundImage else { return nil }

        // crop the foreground based on time
        var foregroundCrop = foregroundImage.extent
        foregroundCrop.size.width *= inputTime
        foregroundCrop.size.height *= inputTime

        return foregroundImage.cropped(to: foregroundCrop).composited(over: backgroundImage)
    }
}

Woosley

Yes, I was aware of this but I wanted to experiment with CIKernel code as I need to write lot of custom transitions. For instance, in this wipe transition I also want to apply a feather on the edges of wipe rectangle. I believe that can't be done using builtin filters. – Chemisorption

It can be done, but it's a bit more complicated. You can "duplicate" the cropped foreground, apply a CIGaussianBlur to it, and composite the original cropped foreground on top of it (so that the actual content doesn't get blurred by the edges do). Then composite that on top of the background. – Woosley

Yes, if the extent of Gaussian blur image is little more than the original image, it would work. What about feather/Gaussian blur along a diagonal line or for that matter, boundary of arbitrary CGPath? How to do that in CIFilter (Metal Core Image Kernel)? – Chemisorption

For that, it's probably best to create some mask image using Core Graphics path drawing APIs and then use CIMaskedVariableBlur directly or CIBlendWithRedMask for blending. – Woosley

That would be problematic to do every frame (30 fps) on older devices that are not so fast. – Chemisorption

Yes, you would need to benchmark performance for sure. But CG path rendering is very fast in my experience and image transfer to GPU is very cheap thanks to shared memory. You can also render the path in a relatively low resolution if you plan to use it for a blurred mask anyways. – Woosley