Shadow catcher

Question

Hi all! I'm trying to do a 2.5D point&click game, where the background is going to be drawn in 2D but the character and the objects are going to be 3D. I would like to have a simple invisible 3D geometry for some of the drawn objects to simulate the shadows: for example, I'll have a 3D mug, a 2D table, and an invisible 3D table whose only goal is to provide the shadow the mug casts onto the table.

I managed to make a simple shader which renders only the shadow and keeps everything transparent otherwise, but the problem is, even if I mark the shadow-catcher material it as "don't cast shadow", the original shadow from the object (mug) is being rendered onto the background (apart from the shadow-catcher).

I know I could turn off the receiving shadows on the background, but then I'd need to split it into the part covered by shadow-catcher and not covered by it (as there will be some parts of the background which should attract shadows), so I wonder:

Is it possible to make a shader/material that would actually prevent the shadows to go through it?

Gloucester

Answer 1

You could play around with the light's cull mask to make it only affect specific objects. However, DirectionalLight cull mask doesn't work in Godot 3.x (but it does in master, where Shadow To Opacity doesn't work correctly yet).

Yeaton

Answer 2

You can write to DEPTH in a fragment shader. If you set it to 1.0, that will make it far back (like part of the sky) and should be far enough away not to display on any objects. Though that may make all shadows invisible. I never tried it, but it's worth looking into.

Paquito

Answer 3

Thank you for the comments. I tried playing with DEPTH a bit and this seems to be going in the right direction, but still I'm not able to do it properly. The problem with the methods that operate on the object level is that I'd like the shadows to be caught only when they go through the shadow-catcher material, but not if the shadow is cast directly on the background. See an example here:

In here, I would like the shadow on the green background to appear only partially (and partially to be caught by the shadow-catcher material in between).

I was thinking I could achieve that by changing the color of the light coming out of the shadow catcher in the shader (in a per-pixel level), and then, on the receiving material change the light back to normal based on whether the light color is modified or not. I tried doing that, but failed to change the color coming out of the material.

Gloucester

Answer 4

Yes, I think I understand. If the invisible objects, like the table, write their DEPTH to the buffer (but have their shadow casting disabled), then this should create a partial shadow.

Paquito

Answer 5

@cybereality said: Yes, I think I understand. If the invisible objects, like the table, write their DEPTH to the buffer (but have their shadow casting disabled), then this should create a partial shadow.

I tried it out, but when I put the DEPTH of the table to 1., it is placed behind the background, and doesn't influence the shadow on the background:

Am I doing something wrong here?

Gloucester

Answer 6

No, not to 1. To the actual depth of the object fragment. I just don't know how the rendering code works (meaning are invisible objects still written to depth and what is written to the depth map)? I could do some tests later today.

Paquito

Answer 7

So I looked into this. It's actually a lot more complex than I thought. One thing I did discover is shadow to opacity, which only renders the shadowed regions and not the rest of the mesh. So this solves the invisible table issue without a custom shader (it's in the material flags).

However, clipping the shadow may be very difficult or impossible. Because the way shadow mapping works (let's say for a directional light to make it simple) it renders all the shadow casting objects into a depth buffer. Then reads from the shadow depth buffer and the screen depth buffer to compare them to see what is blocked. AFAIK it is not possible with this method to render half of an object. An object is either casting a shadow or not. And once it is in the depth map, there is no way to know what object belongs to what, or any way to erase part of an object. So this is tricky.

At one point I was developing this screen space shadow shader. I never got it 100% working, but I did like the idea and years later they started adding it to games (Cyberpunk 2077 uses it for example). It's sort of like ray tracing but in screen space. So you could cast rays from the light, see if they get blocked, and then continue the ray until the next object, if it hits you know that pixel is in a shadow. This is much slower than shadowing mapping, but I think would be fast enough for your game, since you have limited dynamic objects and can limit the angle and coverage of the light.

However, that still doesn't solve how you erase the shadow. We would need a way to mark an object as invisible and or non-shadow casting and then if a ray hit that sort of object (the invisible object) then the ray would be consumed and disappear. This would in effect, allow the cup or whatever to cast shadows on the table, and also partial shadows on the floor (without the table itself casting a shadow). But I need to figure out how to differentiate the invisible objects (since this works in screen space, we only have the color and depth of the final image). Possibly I could write to the alpha channel, I only really need 1-bit of information. I'm not sure if that would mess up the rendering at all, but I should be able to fix it in post.

Though this still depends on being able to solve the whole screen space shadow thing, which I never got 100% working (but I know the idea works, I've seen videos). I can look into it, but it's a more involved project. It may take about 1 week if I wasn't working on other things. Here's a video to show you that it can work.

Not sure what your level of shading programming is, but it's kind of complex. You can look into it if you want, but I wanted to figure it out at some point, so if you can wait a few weeks I can probably get it working.

Paquito

Answer 8

Wow, thanks for the detailed analysis! Now I understand why this is hard… I think: do you know of any docs explaining how does the godot’s rendering process use the depth map? I only found as much:

To compute shadow maps, the scene is rendered (only depth) from an orthogonal point of view that covers the whole scene

Thanks for the offer to take a further look at the screen space shadow casting. I would be eager to play with that, but I don’t have much shader experience (only regular programming one), so realistically I likely won’t be able to be too useful. Let me know if there’s some part you’d be able to delegate; do you have a code started somewhere I could play with?

Gloucester

Answer 9

I would read this page first if you want to know how shadow mapping works (this is OpenGL, but the technique hasn't changed fundamentally since it was invented). Godot and most every engine is based on the same idea. https://learnopengl.com/Advanced-Lighting/Shadows/Shadow-Mapping

The screen space shadow algorithm is here, in a chapter of GPU PRO 6. https://books.google.nl/books?id=30ZOCgAAQBAJ&pg=PA297&lpg=PA297#v=onepage&q&f=false

However, it is pretty advanced and I'm not sure I'd recommend that unless you're already comfortable writing shaders. However, I need this for my next project anyway, so I'll probably start working on it in about 1 or 2 weeks once I finish up what I'm doing now.

Paquito