Looking for help with solution for Sprite3D clipping

Question

Also posted on Reddit. See this thread on the Unity forum for more about the problem and conceptual solution.

When using billboarded sprites in 3D, a common problem without a ready solution in Godot is restricting how the sprite clips with its surroundings. I've been learning shader coding and decided to try tackling this issue. I'm tantalizingly close to getting it, but I'm missing a critical transformation and I haven't been able to track down what it is and where to apply it. I would be grateful if someone could lend me a hand over this hurdle. ?

As for the problem I'm trying to solve, there are two types of unwanted clipping:

Clipping Problem #1: Frequently, a sprite imitating an object in 3D space will have an imaginary 3D origin that may be several pixels above the bottom of the texture. But a Sprite3D cannot appear to rest on the middle of the texture; it exists in 3D space, so it clips through the floor/ground.

Clipping Problem #2: Enabling regular billboard mode on a Sprite3D twists it around in 3D space in a way that is perspective-correct for a 3D quad, but this causes it to clip through nearby 3D meshes. Opting for Y billboard in a bid to prevent this makes the sprite appear flat and squashes its proportions at elevated angles.

Solution: Orient the sprite as a standard billboard, but use a shader to modify its DEPTH value based on two planes: upright like a Y billboard, and flat across XZ. Whichever depth is closer determines DEPTH.

This fixes problem #1 because DEPTH for the pixels that would clip through the floor/ground will instead be based on the XZ plane. A character's foot will appear above the floor, but behind anything higher than the floor where they're standing.

This also fixes problem #2 because DEPTH for the pixels that would clip through a wall will instead be based on the upright plane. A character standing next to a wall can retain the perspective of the source sprite, but remain in front of the wall when looking down from a higher position.

Progress: I've got a recognizable ALBEDO value that can display vertical and lateral planes upon the face of a Sprite3D, proving they're aligned correctly with the point of view. However, the value isn't scaled/curved correctly for DEPTH, causing near values to clip in front of things and far values to rapidly exceed the far clipping distance of the camera. (Once it's working right, I guess the maximum depth should be capped somehow.)

Here's the shader WIP:

shader_type spatial;
render_mode blend_mix, depth_draw_opaque, cull_back;

varying float y_pos;


float depth_to_plane(vec3 dir, vec3 origin, vec3 normal) {
    float denom = dot(normal, dir);
    if (abs(denom) > 1e-5) return dot(origin, normal) / denom;
    else return -1.0;
}


void vertex() {
    // Straight billboard
    //MODELVIEW_MATRIX = INV_CAMERA_MATRIX * mat4(CAMERA_MATRIX[0],CAMERA_MATRIX[1],CAMERA_MATRIX[2],WORLD_MATRIX[3]);
    // Y billboard
    //MODELVIEW_MATRIX = INV_CAMERA_MATRIX * mat4(vec4(normalize(cross(vec3(0.0, 1.0, 0.0), CAMERA_MATRIX[2].xyz)),0.0),vec4(0.0, 1.0, 0.0, 0.0),vec4(normalize(cross(CAMERA_MATRIX[0].xyz, vec3(0.0, 1.0, 0.0))),0.0),WORLD_MATRIX[3]);
    // Straight billboard with Y billboard shading
    MODELVIEW_MATRIX = INV_CAMERA_MATRIX * mat4(CAMERA_MATRIX[0],CAMERA_MATRIX[1],vec4(normalize(cross(CAMERA_MATRIX[0].xyz, vec3(0.0, 1.0, 0.0))),0.0),WORLD_MATRIX[3]);

    y_pos = VERTEX.y;
}


void fragment() {
    // plane origins are both vec3(0.0) in this frame of reference
    vec3 point = CAMERA_MATRIX[3].xyz - WORLD_MATRIX[3].xyz;
    vec3 front_normal = normalize(vec3(point.x, 0.0, point.z));
    // transform from world space to view space
    vec3 point_vs = (INV_CAMERA_MATRIX * vec4(point, 0.0)).xyz;
    front_normal = (INV_CAMERA_MATRIX * vec4(front_normal, 0.0)).xyz;
    vec3 up = (INV_CAMERA_MATRIX * vec4(0.0, 1.0, 0.0, 0.0)).xyz;

    float vert_depth = depth_to_plane(VIEW, point_vs, front_normal);
    float lat_depth = depth_to_plane(VIEW, point_vs, up);
    // abs() or not?
    vert_depth = abs(vert_depth);
    lat_depth = abs(lat_depth);

    float above = float(point.y > 0.0);
    float upper_depth = min(vert_depth, lat_depth) * above + vert_depth * (1.0 - above);
    float depth = float(y_pos >= 0.0) * upper_depth + float(y_pos < 0.0) * min(vert_depth, lat_depth);
    //depth = (0.5 * depth) + 1.0;
    //depth = 2.0 * depth - 1.0;
    //depth = 1.0 / depth;
    ALBEDO = vec3(depth);
    DEPTH = depth;
}

Smooth

Answer 1

You're on the right track. This is totally possible in Godot, and I've done some experiments with writing to depth, though I ended up going with another solution. It's not clear from the images what the object represents. It's important to know what the billboard is of (the shape and outline of the sprite) so you know how to write the depth. For a sphere it is easy. You just use the sphere equation to get the depth from the center, and then discard the pixels outside the sphere. If you are using an alpha channel, then the discard is unneeded and you can just write the max depth since you won't see it anyway. Plus discard is a heavy operation in a pixel shader.

Semipermeable

Answer 2

The sprite in the screenshots is just an icon.png that hasn't had its texture put back on yet -- hiding behind a sphere, over a quad floor. I have a general-purpose depth mapping in mind, as described regarding the two planes. My idea is to center it around Y = 0.0 from the vertex function, which will allow the offset property from SpriteBase3D to determine the clipping behavior by moving the "foot" of the sprite. That seems handy.

In practice, sprites would generally take on the depth of a 90-degree fold, depending on the viewing angle. I've got those depths, in what I thought was view space, but I cannot find where I missed a step (or two) to get the value right for DEPTH. I haven't had luck trying other transformations; ALBEDO indicates they're wrong and DEPTH hasn't gotten any better.

Smooth

Answer 3

Moving the vertices will be difficult, or impossible if the floor is any sort of details (like bumps or uneven terrain). If you know the sprite is always going to be smaller than the grid of the floor, then it might work, but you still have to account for if the sprite is on the edge of two grid squares, in which case it wouldn't be a flat angle. But if you know how your level is setup and carefully place the trees, this may not be an issue. You may want to research how decals work, since it looks similar to what you are attempting. I haven't coded decals from scratch, but it seems like some of the same math would apply.

Another idea is to never move any vertices. You can only write to pixels that are being hit on the polygon. But that polygon can be any shape. For example, you could make the trees cube shapes (sized to the minimum of the size that will encompass the tree) and then do whatever you want in the pixel shader. This is easier since you avoid any heavy vertex math, and you know the cube will always be visible, no matter what object is clipping, since the tree is full encompassed. In the pixel shader, you can get the camera vector and then paint the texture so it is facing the camera.

You'll still need to write the depth, but this is easy, you can generate images in Photoshop or GIMP and just pass them into the shader as a second texture. In terms of the camera angle, it is easiest to pass this as a uniform into the shader. While it is possible to extract this from the model view projection matrix, it is unnecessary math and it will be faster to pass it in.

I've never tried this, but it sounds simple enough. The only hard math would be converting the texture coordinates into the view space, but if it is axis aligned (like always facing forward) this should be straight forward. Let me know if that makes sense, I'm pretty sure that is what you want to do.

Semipermeable

Answer 4

A volume-based approach is an interesting alternative way of considering the problem. I'm not sure whether it is more or less complex than what I've attempted.

I may have misled you with the previous illustration, so here's a better one:

The only vertex processing I'm doing is the blend I created of the default billboard and Y billboard equations (simply copied from converted SpatialMaterial code). In my vertex() function, sprites take the form of a normal billboard in XY view space, with the Z of a Y billboard. It stops the sprite's lighting from changing when the camera moves up and down, which doesn't really make sense.

I'm attempting to leave the vertices where they are, and modify the DEPTH value to take the pixels that I don't want clipped and force them to draw at a desired depth. The plane distance function provides the depths that should enable the concept to work, similar to Ben Golus' Unity shader (link at the top).

Maybe doing something to the [3] vector of the billboard matrix in vertex() will help?

Smooth

Answer 5

Okay. I understand now. Yes, you original idea will work. The moved part of the billboard is fine at at 90 degree angle, provided it is sufficiently above the ground. It can be even offset a bunch and it will still look correct since you are writing the depth manually.

The volume approach would require no vertex math, so that will be easier, but require more pixel math, meaning it will be slower. If you are not drawing fake 3D sprites, then this is probably not performance you want to waste (I assumed the trees were volumes). If you are drawing 2D sprites in 3D, then you don't need this.

But you still don't necessarily need to adjust any vertices. You can use the volume idea with a flat plane. I believe this is the idea behind imposters. Again, this is a fake 3D effect, but the same math should apply in terms of rotating the quad and drawing the pixels in perspective.

Unfortunately, I don't have the time right now to look into the math. But you are on the right track.

Semipermeable

Answer 6

I think I've got the answer I needed for how to correctly modify DEPTH, thanks to this Github issue. Copying the code there, at least now I can write a value to DEPTH without breaking it.

That was a tough search for a couple short lines...

EDIT: ...but it paid off for a big improvement. Here's the origin of our now-textured sprite. The bottom isn't clipping!

It can be occluded correctly (some of the time):

And ta-da! It takes quite an angle to clip it into the sphere if you look at it head-on:

It's not there yet, but it's a start. Parts of the sprite will still disappear at odd angles. Here's the new code to share:

shader_type spatial;
render_mode blend_mix, depth_draw_opaque, cull_back;

uniform sampler2D tex: hint_albedo;

varying float y_pos;


float depth_to_plane(vec3 dir, vec3 origin, vec3 normal) {
	float denom = dot(normal, dir);
	if (abs(denom) > 1e-5) return dot(origin, normal) / denom;
	else return -1.0;
}


void vertex() {
	// Straight billboard
	mat4 normal_billboard = mat4(
		CAMERA_MATRIX[0],
		CAMERA_MATRIX[1],
		CAMERA_MATRIX[2],
		WORLD_MATRIX[3]
	);
	
	// Y billboard
	mat4 y_billboard = mat4(
		vec4(normalize(cross(vec3(0.0, 1.0, 0.0), CAMERA_MATRIX[2].xyz)), 0.0),
		vec4(0.0, 1.0, 0.0, 0.0),
		vec4(normalize(cross(CAMERA_MATRIX[0].xyz, vec3(0.0, 1.0, 0.0))), 0.0),
		WORLD_MATRIX[3]
	);
		
	// Straight billboard with Y billboard shading
	mat4 billboard = mat4(
		CAMERA_MATRIX[0],
		CAMERA_MATRIX[1],
		vec4(normalize(cross(CAMERA_MATRIX[0].xyz, vec3(0.0, 1.0, 0.0))), 0.0),
		WORLD_MATRIX[3]
	);
	
	MODELVIEW_MATRIX = INV_CAMERA_MATRIX * normal_billboard;
	
	y_pos = VERTEX.y;
}


void fragment() {
	// plane origins are both vec3(0.0) in this frame of reference
	vec3 point = CAMERA_MATRIX[3].xyz - WORLD_MATRIX[3].xyz;
	vec3 front_normal = normalize(vec3(point.x, 0.0, point.z));
	// transform all from world space to view space
	vec3 point_vs = (INV_CAMERA_MATRIX * vec4(point, 0.0)).xyz;
	front_normal = (INV_CAMERA_MATRIX * vec4(front_normal, 0.0)).xyz;
	vec3 up = (INV_CAMERA_MATRIX * vec4(vec3(0.0, 1.0, 0.0), 0.0)).xyz;
	
	float vert_depth = depth_to_plane(VIEW, point_vs, front_normal);
	float lat_depth = depth_to_plane(VIEW, point_vs, up);
	// abs() or not?
	vert_depth = abs(vert_depth);
	lat_depth = abs(lat_depth);
	
	// magic margin on lateral depth
	lat_depth -= 0.1;
	
	float above = float(point.y > 0.0);
	float upper_depth = min(vert_depth, lat_depth) * above + vert_depth * (1.0 - above);
	float depth = float(y_pos >= 0.0) * upper_depth + float(y_pos < 0.0) * min(vert_depth, lat_depth);
	
	vec4 clip = PROJECTION_MATRIX * vec4(VERTEX.xy, -depth, 1.0);
	DEPTH = 0.5 * clip.z / clip.w + 0.5;
	ALBEDO = texture(tex, UV).rgb;
}

As complex as this is proving to be, for now, I think I'm going to go back to exploring what I can achieve with literally 2D Sprites, because Sprite3D just does not get the job done for me. For one thing, I hadn't considered the ShaderMaterial breaking so much of the Sprite3D functionality. :/

Smooth

Answer 7

Welp. I had an inspiring breakthrough with the 2D-based project, yielding an ~80% FPS boost, way in excess of what I should really worry about anyway ("1-2ms per frame? MUST OPTIMIZE MORE!" Just kidding.). =)

Fair notice for anyone lurking hoping I'd keep chipping away at Sprite3D -- I may return to it, but I've got other things to try.

Smooth

Answer 8

Hi Wolfe,

Your shader works perfectly for my game's use case. Would you mind if I use it (and giving you credit of course)?

Also I added support for the alpha channel with adding the following at the end of the shader:
ALPHA = texture(tex, UV).a;

Lamberto

Answer 9

Go for it! 👍️

Smooth

Answer 10

Hi Wolve,

The shader works for sprites that are centered on camera, but bugs for sprites far away from the center of camera as you can see in my screenshot. Do you have any clue on how to solve this?

Below is the shader I'm using. I have tweaked it a little in order to get it working on Godot 4.x, but my knowledge on shaders is very limited so I have absolutely no idea on how to solve this.

`shader_type spatial;
render_mode blend_mix, depth_draw_opaque, cull_back;

uniform sampler2D tex: source_color;

varying float y_pos;

float depth_to_plane(vec3 dir, vec3 origin, vec3 normal) {
float denom = dot(normal, dir);
if (abs(denom) > 1e-5) return dot(origin, normal) / denom;
else return -1.0;
}

void vertex() {
// Straight billboard
mat4 normal_billboard = mat4(
INV_VIEW_MATRIX[0],
INV_VIEW_MATRIX[1],
INV_VIEW_MATRIX[2],
MODEL_MATRIX[3]
);

// Y billboard
mat4 y_billboard = mat4(
	vec4(normalize(cross(vec3(0.0, 1.0, 0.0), INV_VIEW_MATRIX[2].xyz)), 0.0),
	vec4(0.0, 1.0, 0.0, 0.0),
	vec4(normalize(cross(INV_VIEW_MATRIX[0].xyz, vec3(0.0, 1.0, 0.0))), 0.0),
	MODEL_MATRIX[3]
);
	
// Straight billboard with Y billboard shading
mat4 billboard = mat4(
	INV_VIEW_MATRIX[0],
	INV_VIEW_MATRIX[1],
	vec4(normalize(cross(INV_VIEW_MATRIX[0].xyz, vec3(0.0, 1.0, 0.0))), 0.0),
	MODEL_MATRIX[3]
);

MODELVIEW_MATRIX = VIEW_MATRIX * normal_billboard;

y_pos = VERTEX.y;

}

void fragment() {
// plane origins are both vec3(0.0) in this frame of reference
vec3 point = INV_VIEW_MATRIX[3].xyz - MODEL_MATRIX[3].xyz;
vec3 front_normal = normalize(vec3(point.x, 0.0, point.z));
// transform all from world space to view space
vec3 point_vs = (VIEW_MATRIX * vec4(point, 0.0)).xyz;
front_normal = (VIEW_MATRIX * vec4(front_normal, 0.0)).xyz;
vec3 up = (VIEW_MATRIX * vec4(vec3(0.0, 1.0, 0.0), 0.0)).xyz;

float vert_depth = depth_to_plane(VIEW, point_vs, front_normal);
float lat_depth = depth_to_plane(VIEW, point_vs, up);
// abs() or not?
vert_depth = abs(vert_depth);
lat_depth = abs(lat_depth);

// magic margin on lateral depth
lat_depth -= 0.1;

float above = float(point.y > 0.0);
float upper_depth = min(vert_depth, lat_depth) * above + vert_depth * (0.5 - above);
float depth = float(y_pos >= 0.0) * upper_depth + float(y_pos < 0.0) * min(vert_depth, lat_depth);

vec4 clip = PROJECTION_MATRIX * vec4(VERTEX.xy, -depth, 1.0);
DEPTH = 0.5 * clip.z / clip.w + 0.5;
ALBEDO = texture(tex, UV).rgb;
ALPHA = texture(tex, UV).a;

}
`

Lamberto