How to make fragment shader consider lights

Question

I have an idea for a method of faking 3D shadows in 2D, but I'm struggling to implement it due to my lack of experience. I want to achieve this by manipulating sprites with a custom shader.

TL;DR

I need a way for a CanvasItem shader to get the colour values of a sprite after a light has affected it, not before. CanvasGroups don't work for me. How could I achieve this?

My Plan

Each sprite to be shaded is actually constructed from two: the sprite when it's completely lit, and the sprite when it's in shade. It uses its channels in a special way to combine these and other information. Colours aren't stored directly but indexed to a palette.

R — Index values referencing a palette for when the sprite is Lit
G — Index values referencing a palette for when the sprite is in Shade
B — Height value for each pixel
A — Alpha

I have exaggerated the values for of the Lit, Shade and Height components to be more visible. They'd really be very dark.

A shader will use a palette texture (as a uniform) to colour in the sprite considering which parts are lit (it will use R) or in shade (it will use G). It will determine this by looking at the blue channel (height).

I want the fragment shader to look at the blue component and if it is zero or less the the green channel is used instead of the red channel.

How shadows are achieved:

1. Box with a ground shadow. Sun is from top left.

2. The shade the box casts can be thought of as a volume, a triangular prism.

3. Heightmap of this shade volume in blue channel (values exaggerated)

4. Shade heightmap blend mode set to subtract over character heightmap. Where this results in the players blue channel equalling zero there should be shade.

5. Sprite shader considers the blue channel and shades sprite accordingly.

Except I can't figure out how to have the sprite shader consider the Light2D properly. The fragment shader can't consider the effects of lights, because it's just acting on the sprite texture. I know there is a light function as well as the fragment function for CanvasItem shaders, but idk how to use this. I tried a CanvasGroup over the sprite which I then applied a shader to, but CanvasGroups don't have a texture accessible to shaders, leading to a white box.

Cordula

Answer 1

From my experiments in beta 2, in the light() shader function, you get the inout LIGHT parameter -- that's your color after the light had affected the sprite. Try putting your calculations in the light() function (and make sure the shader is not unshaded).

Abdullah

Answer 2

I've tried using the LIGHT built-in but it doesn't seem to do anything? Also, from my fiddling the LIGHT built-in it seems to be the output value for the light, not for the sprite's appearance after lights have been applied. When I try access LIGHT it behaves unexpectedly by doing nothing.

This:

void fragment(){
    COLOR = COLOR;
}

changes nothing, as this is the default. Meanwhile,

void light(){
    LIGHT = LIGHT;
}

completely negates any lights as if they weren't there, which is not the default. So confusing.

I did have luck with using LIGHT_COLOR though. You'd expect it to fetch the color parameter of the light like the documentation implies, but no, it samples the light's colour overall (including the texture) like the LIGHT built-in should.

Your reply was still really useful — I managed to do what I wanted by computing in the light shader instead. Thanks!
: D

Cordula