Reverse a CALayer mask

Question

I am trying to use a CALayer with an image as contents for masking a UIView. For the mask I have complex png image. If I apply the image as a view.layer.mask I get the opposite behaviour of what I want. Is there a way to reverse the CAlayer? Here is my code:

layerMask = CALayer()
guard let layerMask = layerMask else { return }    
layerMask.contents = #imageLiteral(resourceName: "mask").cgImage
view.layer.mask = layerMask
// What I would like to to is
view.layer.mask = layerMask.inverse. // <---

I have seen several posts on reverse CAShapeLayers and Mutable paths, but nothing where I can reverse a CALayer. What I could do is reverse the image in Photoshop so that the alpha is inverted, but the problem with that is that I won't be able to create an image with the exact size to fit all screen sizes. I hope it does make sense.

Corunna

The problem with that is that the mask has a fixed size in the center of the key window. If I would draw the "correct" mask in Photoshop, I will never get the mask to work on all screen sizes. So I using my company logo as the mask, which I need to stencil out from the main view – Corunna

No, I don't follow. Why would that work if you start with your mask but not work if you start with the correct mask? – Flattie

Inverse your mask in Photoshop. Else you can invert it in code, but why would you do that over Photoshop? – Kathaleenkatharevusa

I have added an example image to the main post – Corunna

Well, my answer tells you how to do it. If you start with that triangly thing drawn in black, you can make an image of any desired size containing the triangly thing and turn it into a mask that does exactly what you want. – Flattie

Okay! Thanks Matt. Sorry for being so noobish about it, but could you update your answer with some code examples please? – Corunna

I don't know precisely what you want to do (e.g. how you want to place the logo etc.). And I don't have your images, your logo, etc. All I can do is tell you the technique, and I believe I've done that. I have added screen shots showing the stages to illustrate what I said. – Flattie

This is one of Core Animation's more frustrating limitations. It really just needs an invertsMask property to reverse the pixel alpha values for cases like this. – Boeotia

Answer 1

What I would do is construct the mask in real time. This is easy if you have a black image of the logo. Using standard techniques, you can draw the logo image into an image that you construct in real time, so that you are in charge of the size of the image and the size and placement of logo within it. Using a "Mask To Alpha" CIFilter, you can then convert the black to transparent for use as a layer mask.

So, to illustrate. Here's the background image: this is what we want to see wherever we punch a hole in the foreground:

Here's the foreground image, lying on top of the background and completely hiding it:

Here's the logo, in black (ignore the grey, which represents transparency):

Here's the logo drawn in code into a white background of the correct size:

And finally, here's that same image converted into a mask with the Mask To Alpha CIFilter and attached to the foreground image view as its mask:

Okay, I could have chosen my images a little better, but this is what I had lying around. You can see that wherever there was black in the logo, we are punching a hole in the foreground image and seeing the background image, which I believe is exactly what you said you wanted to do.

The key step is the last one, namely the conversion of the black-on-white image of the logo (im) to a mask; here's how I did that:

    let cim = CIImage(image:im)
    let filter = CIFilter(name:"CIMaskToAlpha")!
    filter.setValue(cim, forKey: "inputImage")
    let out = filter.outputImage!
    let cgim = CIContext().createCGImage(out, from: out.extent)
    let lay = CALayer()
    lay.frame = self.iv.bounds
    lay.contents = cgim
    self.iv.layer.mask = lay

Flattie

Yes thats it! Thank a lot! I just need to get the code to work. Will post it when I get it to work. Thanks for your time! – Corunna

The key part is the conversion of the black-on-white image to a transparency mask, and I can certainly give you that; hang on a sec. – Flattie

There you go. I've added the code for the final step, which is the part I would assume you might not know how to do. – Flattie

Great!! Thats it :) I found a similar thing with C ..but was struggling. Thanks again Matt – Corunna

dude, you should be more declarative within those variables naming – Occidental

I agree with @FericoSamuel. Those are some really awful variable names. Is this K&R C or something? :) – Coral

@DuncanC Everybody's a critic! – Flattie

So CIMaskToAlpha makes black pixels transparent in the resulting alpha, white pixels opaque (makes sense) but also makes transparent source pixels opaque? (I was not aware of that last bit.) – Coral

@DuncanC I didn't say any such thing. – Flattie

I'm trying to understand why your code works. Can you add some explanation? – Coral

Ok, I think I understand what you're saying now. You have to have an intermediate step where you render your black-on-transparent image onto a white background in order to create the black-on-white image that is fed into CIMaskToAlpha? – Coral

@DuncanC I believe that is what the answer says, very clearly, step by step. – Flattie

This answer is very confusing, because you say "Here's the logo, in black (ignore the grey, which represents transparency)" when in fact, to achieve the desired result, the mask image should be completely opaque and only have black color (mask out) and white color (keep, dont mask), no transparent pixels. – Covet

@Flattie there is a youtube video with your famous solution already 😀youtube.com/watch?v=muUOLQxFB2g – Tinney

@Flattie could you look at my example https://mcmap.net/q/750479/-how-to-make-a-transparent-hole-in-uivisualeffectview/2725435? I couldnt apply solution for image to my case with rectangle. – Tinney

Answer 2

If you're using a CALayer as a mask for another CALayer, you can invert the mask by creating a large opaque layer and subtracting out the mask shape with the xor blend mode.

For example, this code subtracts a given layer from a large opaque layer to create an mask layer:

// Create a large opaque layer to serve as the inverted mask
let largeOpaqueLayer = CALayer()
largeOpaqueLayer.bounds = CGRect(x: -10_000_000, y: -10_000_000, width: 20_000_000, height: 20_000_000)
largeOpaqueLayer.backgroundColor = UIColor.black.cgColor

// Subtract out the mask shape using the `xor` blend mode
let maskLayer = ...
largeOpaqueLayer.addSublayer(maskLayer)
maskLayer.compositingFilter = "xor"

Then you can use that layer as the mask for some other CALayer. For example here I'm using it as the mask of a small blue rectangle:

smallBlueRectangle.mask = largeOpaqueLayer

So you can see the mask is inverted! On the other hand if you just use the un-inverted maskLayer directly as a mask, you can see the mask is not inverted:

Matchbox

Great answer. Just what I was looking for. – Paleoasiatic

Is this property supported on iOS? developer.apple.com/documentation/quartzcore/calayer/… Also weird that you set a property to a string "xor" -- how are we supposed to know what the possible keys are? (does this work for Objective-C?) – Unwinking