Smoothen Edges of Chroma Key -CoreImage

Question

I'm using the following code to remove green background from an image.But the edges of the image has green tint to it and some pixels are damaged.How can i smoothen this and make the cut out perfect.

 func chromaKeyFilter(fromHue: CGFloat, toHue: CGFloat) -> CIFilter?
    {
        // 1
        let size = 64
        var cubeRGB = [Float]()

        // 2
        for z in 0 ..< size {
            let blue = CGFloat(z) / CGFloat(size-1)
            for y in 0 ..< size {
                let green = CGFloat(y) / CGFloat(size-1)
                for x in 0 ..< size {
                    let red = CGFloat(x) / CGFloat(size-1)

                    // 3
                    let hue = getHue(red: red, green: green, blue: blue)
                    let alpha: CGFloat = (hue >= fromHue && hue <= toHue) ? 0: 1

                    // 4
                    cubeRGB.append(Float(red * alpha))
                    cubeRGB.append(Float(green * alpha))
                    cubeRGB.append(Float(blue * alpha))
                    cubeRGB.append(Float(alpha))
                }
            }
        }



  @IBAction func clicked(_ sender: Any) {
          let a = URL(fileURLWithPath:"green.png")
          let b = URL(fileURLWithPath:"back.jpg")

        let image1 = CIImage(contentsOf: a)
        let image2 = CIImage(contentsOf: b)





        let chromaCIFilter = self.chromaKeyFilter(fromHue: 0.3, toHue: 0.4)
        chromaCIFilter?.setValue(image1, forKey: kCIInputImageKey)
        let sourceCIImageWithoutBackground = chromaCIFilter?.outputImage

        /*let compositor = CIFilter(name:"CISourceOverCompositing")
        compositor?.setValue(sourceCIImageWithoutBackground, forKey: kCIInputImageKey)
        compositor?.setValue(image2, forKey: kCIInputBackgroundImageKey)
        let compositedCIImage = compositor?.outputImage*/

        var rep: NSCIImageRep = NSCIImageRep(ciImage: sourceCIImageWithoutBackground!)
        var nsImage: NSImage = NSImage(size: rep.size)
        nsImage.addRepresentation(rep)

        let url = URL(fileURLWithPath:"file.png")

        nsImage.pngWrite(to: url)
        super.viewDidLoad()
    }

Input:

Output:

Update:

Update 2:

Update 3:

Roslyn

Why did you comment out that code? I see that it is the default code from Apple's example, developer.apple.com/documentation/coreimage/…, so what made you change their example if it works? – Blankenship

@Blankenship As far as i understand that piece of code replaced the background image.But i just need to remove the background.The apple sample code is not for macOS , i have used CIImage while the OSX sample uses UIImage. – Roslyn

Ahh i see. I was just curious. – Blankenship

Hey @techno, did you solve your problem? I am trying to modify the default sample app from apples doc, but I am facing some issues with removing the color. Would you be so kind and would check my question? Thanks a lot dude #74231229 – Thorathoracic

Answer 1

Professional tools for chroma keying usually include what's called a spill suppressor. A spill suppressor finds pixels that contain small amounts of the chroma key color and shifts the color in the opposite direction. So green pixels will move towards magenta. This reduces the green fringing you often see around keyed footage.

The pixels you call damaged are just pixels that had some level of the chroma color in them and are being picked up by your keyer function. Rather than choosing a hard 0 or 1, you might consider a function that returns a value between 0 and 1 based on the color of the pixel. For example, you could find the angular distance of the current pixel's hue to the fromHue and toHue and maybe do something like this:

// Get the distance from the edges of the range, and convert to be between 0 and 1
var distance: CGFloat
if (fromHue <= hue) && (hue <= toHue) {
    distance = min(abs(hue - fromHue), abs(hue - toHue)) / ((toHue - fromHue) / 2.0)
} else {
    distance = 0.0
}
distance = 1.0 - distance
let alpha = sin(.pi * distance - .pi / 2.0) * 0.5 + 0.5

That will give you a smooth variation from the edges of the range to the center of the range. (Note that I've left off dealing with the fact that hue wraps around at 360°. That's something you'll have to handle.) The graph of the falloff looks like this:

Another thing you can do is limit the keying to only affect pixels where the saturation is above some threshold and the value is above some threshold. For very dark and/or unsaturated colors, you probably don't want to key it out. I think that would help with the issues you're seeing with the model's jacket, for example.

Augusto

seems complex... im just a beginner in swift.. How can i implement this in my code? – Roslyn

I think he told you .. right after your let hue = ... add his code – Blankenship

Yes, sorry, that's exactly what I meant. Note that this was written off the top of my head, so might need some cleanup to have valid syntax. – Augusto

@Augusto Your code produces wrong results.Please see the update. – Roslyn

Sorry about that! I forgot to invert the distance before calculating the alpha. I think I also reversed the fromHue and toHue in the denominator of the distance calculation. I've updated the code. Give it a try and let me know if it works. – Augusto

This also results in problems.Please see the update. – Roslyn

That's actually looking pretty good. Now you just need to adjust the curve so that it takes out the green completely. You could try something like having an and inner from and to value, where everything between those values is full transparent, but between the inner and outer "to" and the inner and outer "from" you use the smooth falloff shown above. – Augusto

Since you're using .3-.4 for your range of hue, you probably actually want .333... to be in the center of the range for this. Try making your range be .3 to .366.... Regardless, having a wide inner area that's fully transparent and a smooth fall-off at the edges is still a good idea. – Augusto

This also does not solve the core problem i have mentioned.Please see the update. – Roslyn

Not 100% related to this question. But I am facing issue with the default sample implementation that you find on apple website. Would you be so kind and check my question @Augusto , please? #74231229 – Thorathoracic

Answer 2

My (live) keyer works like this (with the enhancements user1118321 describes) and using its analyser I quickly noticed this is most likely not a true green screen image. It's one of many fake ones where the green screen seems to have been replaced with a saturated monochrome green. Though this may look nice, it introduces artefacts where the keyed subject (with fringes of the originally used green) meets the monochrome green. You can see a single green was used by looking at the histogram. Real green screen always have (in)visible shades of green. I was able to get a decent key but had to manually tweak some settings. With a NLE you can probably get much better results, but they will also be a lot more complex. So to get back to your issue, your code probably works as it is now (update #3), you just have to use a proper real life green screen image.

Hertha

hmm... but applications like PhotoKey8 works fine with the image i have tested. – Roslyn

If your code does not have to be fast then you either have to buy the technology (the good ones are patented btw) or invest a few years of your life. It's a fascinating technology so enjoy! – Hertha

@Summit I use two: one works in YUV domain and one in RGB, they complement each other. The idea is to suppress the key colour in the foreground and at the mask border. The real trick is to automate this so it works on most images and, in my case where live video is involved, make it fast. There's enough public information, like research papers, on the internet to learn this wonderful technology. – Hertha

@Summit if your suppressor doesn't work then maybe your keyer doesn't work properly? Sorry but I cannot share my code with you, it is part of my commercial product. – Hertha