Metal Shading language for Core Image color kernel, how to pass an array of float3

Question

I'm trying to port some CIFilter from this source by using metal shading language for Core Image.
I have a palette of color composed by an array of RGB struct and I want to pass them as an argument to a custom CI color image kernel.
The RGB struct is converted into an array of SIMD3<Float>.

 static func SIMD3Palette(_ palette: [RGB]) -> [SIMD3<Float>] {
        return palette.map{$0.toFloat3()}
    }

The kernel should take and array of simd_float3 values, the problem is the when I launch the filter it tells me that the argument at index 1 is expecting an NSData.

override var outputImage: CIImage? {
        guard let inputImage = inputImage else
        {
            return nil
        }
         let palette = EightBitColorFilter.palettes[Int(inputPaletteIndex)]
        let extent = inputImage.extent
        let arguments = [inputImage, palette, Float(palette.count)] as [Any]

        let final = colorKernel.apply(extent: extent, arguments: arguments)

        return final
    }

This is the kernel:

float4 eight_bit(sample_t image, simd_float3 palette[], float paletteSize, destination dest) {
        float dist = distance(image.rgb, palette[0]);
        float3 returnColor = palette[0];
        for (int i = 1; i < floor(paletteSize); ++i) {
            float tempDist = distance(image.rgb, palette[i]);
            if (tempDist < dist) {
                dist = tempDist;
                returnColor = palette[i];
            }
        }
        return float4(returnColor, 1);
    }

I'm wondering how can I pass a data buffer to the kernel since converting it into an NSData seems not enough.
I saw some example but they are using "full" shading language that is not available for Core Image that is a sort of subset for dealing only with fragments.

Loosetongued

Answer 1

Update

We have now figured out how to pass data buffers directly into Core Image kernels. Using a CIImage as described below is not needed, but still possible.

Assuming that you have your raw data as an NSData, you can just pass it to the kernel on invocation:

kernel.apply(..., arguments: [data, ...])

Note: Data might also work, but I know that NSData is an argument type that allows Core Image to cache filter results based on input arguments. So when in doubt, better cast to NSData.

Then in the kernel function, you only need to declare the parameter with an appropriate constant type:

extern "C" float4 myKernel(constant float3 data[], ...) {
    float3 data0 = data[0];
    // ...
}

---

Previous Answer

Core Image kernels don't seem to support pointer or array parameter types. Though there seem to be something coming with iOS 13. From the Release Notes:

Metal CIKernel instances support arguments with arbitrarily structured data.

But, as so often with Core Image, there seem to be no further documentation for that…

However, you can still use the "old way" of passing buffer data by wrapping it in a CIImage and sampling it in the kernel. For example:

    let array: [Float] = [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]
    let data = array.withUnsafeBufferPointer { Data(buffer: $0) }
    let dataImage = CIImage(bitmapData: data, bytesPerRow: data.count, size: CGSize(width: array.count/4, height: 1), format: .RGBAf, colorSpace: nil)

Note that there is no CIFormat for 3-channel images since the GPU doesn't support those. So you either have to use single-channel .Rf and re-pack the values inside your kernel to float3 again, or add some strides to your data and use .RGBAf and float4 respectively (which I'd recommend since it reduces texture fetches).

When you pass that image into your kernel, you probably want to set the sampling mode to nearest, otherwise you might get interpolated values when sampling between two pixels:

kernel.apply(..., arguments: [dataImage.samplingNearest(), ...])

In your (Metal) kernel, you can assess the data as you would with a normal input image via a sampler:

extern "C" float4 myKernel(coreimage::sampler data, ...) {
    float4 data0 = data.sample(data.transform(float2(0.5, 0.5))); // data[0]
    float4 data1 = data.sample(data.transform(float2(1.5, 0.5))); // data[1]
    // ...
}

Note that I added 0.5 to the coordinates so that they point in the middle of a pixel in the data image to avoid ambiguity and interpolation.

Also note that pixel values you get from a sampler always have 4 channels. So even when you are creating your data image with formate .Rf, you'll get a float4 when sampling it (the other values are filled with 0.0 for G and B and 1.0 for alpha). In this case, you can just do

float data0 = data.sample(data.transform(float2(0.5, 0.5))).x;

Edit

I previously forgot to transform the sample coordinate from absolute pixel space (where (0.5, 0.5) would be the middle of the first pixel) to relative sampler space (where (0.5, 0.5) would be the middle of the whole buffer). It's fixed now.

Chef

Thank you Frank, I will look into that, for the moment +1 – Loosetongued

@frank-schlegel Could you develop how to receive and manipulate the dataImage on the other side in the metal shader? Which type should we used in the shader for it? – Jeconiah

@Jeconiah I extended my answer. 🙂 – Chef

Awesome 🙏 It helps a lot noobs like me. – Jeconiah

@FrankSchlegel Can you take a look at this question - #70109351 – Hecht

@FrankRupprecht, when sampling the first data point, I'm only able to do so if I use negative values: float4 data0 = data.sample(float2(-0.5, -0.5)); Am I doing something wrong, or did something change in Core Image since your answer? I'm using your technique exactly as explained. – Samba

@Samba Uh, I forgot to add the transform from pixel space to sampler space. Please see my edit above. – Chef

@Samba I also realized that we have since figured out how to properly pass data to a kernel, so I updated my answer with our findings. – Chef

Answer 2

I made it, event if the answer was good and also deploys to lower target the result wasn't exactly what I was expecting. The difference between the original kernel written as a string and the above method to create an image to be used as a source of data were kind of big.
Didn't get exactly the reason, but the image I was passing as a source of the palette was kind of different from the created one in size and color(probably due to color spaces).
Since there was no documentation about this statement:

Metal CIKernel instances support arguments with arbitrarily structured data.

I tried a lot in my spare time and came up to this.
First the shader:

   float4 eight_bit_buffer(sampler image, constant simd_float3 palette[], float paletteSize, destination dest) {
        float4 color = image.sample(image.transform(dest.coord()));

        float dist = distance(color.rgb, palette[0]);
        float3 returnColor = palette[0];
        for (int i = 1; i < floor(paletteSize); ++i) {
            float tempDist = distance(color.rgb, palette[i]);
            if (tempDist < dist) {
                dist = tempDist;
                returnColor = palette[i];
            }
        }
        return float4(returnColor, 1);
    }

Second the palette transformation into SIMD3<Float>:

 static func toSIMD3Buffer(from palette: [RGB]) -> Data {
        var simd3Palette = SIMD3Palette(palette)
        let size = MemoryLayout<SIMD3<Float>>.size
        let count = palette.count * size
        let palettePointer = UnsafeMutableRawPointer.allocate(
            byteCount: simd3Palette.count * MemoryLayout<SIMD3<Float>>.stride,
            alignment: MemoryLayout<SIMD3<Float>>.alignment)
        let simd3Pointer = simd3Palette.withUnsafeMutableBufferPointer { (buffer) -> UnsafeMutablePointer<SIMD3<Float>> in
            let p = palettePointer.initializeMemory(as: SIMD3<Float>.self,
            from: buffer.baseAddress!,
            count: buffer.count)
            return p
        }
        let data = Data(bytesNoCopy: simd3Pointer, count: count * MemoryLayout<SIMD3<Float>>.stride, deallocator: .free)

        return data
    }

The first time I tried by appending SIMD3 to the Data object but wasn't working probably due to memory alignment. Remember to dealloc the memory created after you used it.
Hope to help someone else.

Loosetongued