Using vector colors as normal map for 3D object

Question

I have a 3D model in blender with custom normals that I baked onto the vertex colors. In godot, I'm trying to use the vertex colors as normal for my toon shader to get stylized shadowing on the face. However, the lighting seems borked with the shader I'm using. The shader code is below:

>! shader_type spatial;
>! 
>! #define USE_RIM 1
>! 
>! #define USE_PIXELART 1
>! #define USE_SMOOTH_PIXELART 1
>! 
>! #define USE_ALPHA 1
>! #define USE_DISABLE_CULL 1
>! 
>! #define USE_SUBSURFACESCATTERING 1
>! 
>! render_mode 
>! 	#if USE_DISABLE_CULL
>! 		cull_disabled;
>! 	#else
>! 		cull_back;
>! 	#endif
>! 
>! group_uniforms Toon;
>! uniform float     ToonRampOffset:     hint_range(0.0, 1.0) = 0.5;
>! uniform float     ToonRampSmoothness: hint_range(0.0, 1.0) = 0.05;
>! uniform vec3      ToonRampTinting:    source_color;
>! 
>! #if USE_RIM
>! 	group_uniforms Rim;
>! 	uniform float     RimPower:           hint_range(0.0, 10.0) = 1.0;
>! 	uniform float     RimCutOff:          hint_range(0.0, 1.0) = 0.5;
>! 	uniform float     RimSmoothness:      hint_range(0.0, 1.0) = 0.05;
>! 	uniform float     RimLightBrightness: hint_range(0.0, 50.0) = 20.0;
>! #endif
>! 
>! #if USE_SUBSURFACESCATTERING
>! 	group_uniforms SubsurfaceScattering;
>! 	uniform float     SubSurfDistortion:  hint_range(0.0, 5.0) = 1.0;
>! 	uniform vec3      SubSurfTint:        source_color;
>! 	uniform float     SubSurfBrightness:  hint_range(0.0, 10.0) = 3.0;
>! 	uniform float     SubSurfCutoff:      hint_range(0.0, 1.0) = 0.5;
>! 	uniform float     SubSurfSmoothness:  hint_range(0.0, 1.0) = 0.05;
>! 	uniform sampler2D SubSurfTexture:  source_color, hint_default_white;
>! 	
>! #endif
>! 
>! group_uniforms Texture;
>! #if (USE_SMOOTH_PIXELART || !USE_PIXELART)
>! 	uniform sampler2D Texture:            source_color, filter_linear_mipmap_anisotropic;
>! #else 
>! 	uniform sampler2D Texture:            source_color, filter_nearest_mipmap_anisotropic;
>! #endif
>! 
>! #if USE_RIM
>! 	float fresnel(float amount, vec3 normal, vec3 view) {
>! 		return pow((1.0 - clamp(dot(normalize(normal), normalize(view)), 0.0, 1.0 )), amount);
>! 	}
>! #endif
>! 
>! #if USE_SMOOTH_PIXELART
>! 	vec4 texture_point_smooth(sampler2D smp, vec2 uv, vec2 pixel_size) {
>! 		vec2 ddx = dFdx(uv);
>! 		vec2 ddy = dFdy(uv);
>! 		vec2 lxy = sqrt(ddx * ddx + ddy * ddy);
>! 		
>! 		vec2 uv_pixels = uv / pixel_size;
>! 		
>! 		vec2 uv_pixels_floor = round(uv_pixels) - vec2(0.5f);
>! 		vec2 uv_dxy_pixels = uv_pixels - uv_pixels_floor;
>! 		
>! 		uv_dxy_pixels = clamp((uv_dxy_pixels - vec2(0.5f)) * pixel_size / lxy + vec2(0.5f), 0.0f, 1.0f);
>! 		
>! 		uv = uv_pixels_floor * pixel_size;
>! 		
>! 		return textureGrad(smp, uv + uv_dxy_pixels * pixel_size, ddx, ddy);
>! 	}
>! #endif
>! void vertex () {
>! 	//set range from 0..1 to -1..1
>! 	vec3 normal = COLOR.rgb * 2.0 - 1.0;
>! 	NORMAL = normal;
>! }
>! void fragment() {
>! 	vec4 textureColor;
>! 	
>! 	#if USE_SMOOTH_PIXELART
>! 		vec2 tex_size = 1.0f / vec2(textureSize(Texture, 0));
>! 		textureColor = texture_point_smooth(Texture, UV, tex_size);
>! 	#else
>! 		textureColor = texture(Texture, UV);
>! 	#endif
>! 		ALBEDO = textureColor.rgb;
>! 
>! 	#if USE_ALPHA
>! 		ALPHA = textureColor.a;
>! 		ALPHA_SCISSOR_THRESHOLD = 0.5;
>! 	#endif
>! }
>! 
>! void light() {
>! 	// Diffuse Shading
>! 	float d =  dot(NORMAL,LIGHT) * 0.5 + 0.5;
>! 	float toonRamp = smoothstep(ToonRampOffset, ToonRampOffset + ToonRampSmoothness, d);
>! 	
>! 	// Shadows
>! 	toonRamp *= ATTENUATION;
>! 	
>! 	vec3 toonRampOutput = LIGHT_COLOR * toonRamp;
>! 	vec3 ambientLightOutput = ALBEDO * ToonRampTinting;
>! 	DIFFUSE_LIGHT += clamp((toonRampOutput - ambientLightOutput), vec3(0.0), vec3(1.0));
>! 	
>! 	
>! 	#if USE_RIM
>! 		float fresnel = fresnel(RimPower, NORMAL, VIEW);
>! 		float d2 = dot(NORMAL, LIGHT);
>! 		d2 *= fresnel;
>! 		d2 = smoothstep(RimCutOff, RimCutOff + RimSmoothness, d2);
>! 		
>! 		vec3 rimOutput =  d2 * RimLightBrightness * ALBEDO;
>! 		
>! 		DIFFUSE_LIGHT += rimOutput;
>! 	#endif
>! 	
>! 	
>! 	
>! 	#if USE_SUBSURFACESCATTERING
>! 		float subSurfBack = dot(VIEW, -(LIGHT + NORMAL * SubSurfDistortion));
>! 		float subSurfFront = dot(VIEW, LIGHT - NORMAL * SubSurfDistortion);
>! 		subSurfBack = max(subSurfBack, 0.0);
>! 		subSurfFront = max(subSurfFront, 0.0);
>! 		
>! 		float subSurfScattering = smoothstep(SubSurfCutoff, SubSurfCutoff + SubSurfSmoothness, subSurfBack + subSurfFront); 
>! 		
>! 		vec3 subSurfColor = SubSurfTint * texture(SubSurfTexture, UV).rgb;
>! 		DIFFUSE_LIGHT += subSurfScattering * SubSurfBrightness * subSurfColor;
>! 	#endif         
>!           
>! 	
>! 	SPECULAR_LIGHT = ambientLightOutput;
>! }

The base code is from this shader: https://godotshaders.com/shader/toon-shader-for-godot-4/. I only added the below lines:

void vertex () {
    vec3 normal = COLOR.rgb * 2.0 - 1.0;
    NORMAL = normal;
}

Below is a screenshot. As you can see the shading when changing direction along the x-axis rotation is wrong. Rotation along y-axis seems to be working fine though.

Latonya

Answer 1

Your baked normals are ok. They appear to be the same in Godot and in Blender. The problem (so to speak) seems to be with the first model (named Cube). It looks like there's a difference in up axis convention between baked and actual normals. So if you only plan to use baked normals, you're fine. Otherwise try to disable y+up option in Blender's glTF export options when exporting the non-baked model. You'll have to rotate the model in Godot, but the normals should be the same as in baked model.

Btw, normal textures could not be baked because there are no proper UVs on the model.

Frangos

Answer 2

Can anyone help?

Latonya

Answer 3

Output the vertex color to unshaded albedo first to see if it's looking ok. Vertex shader expects normals to be in local space. Make sure you haven't baked them in tangent or world space.

Frangos

Answer 4

Here's how it looks in Godot 4.0 with vertex color output to albedo and no light.

Here's how it look inside Blender with the custom normals from another model painted on.

I believe my issue is with correctly mapping the colors to normal in Godot 4.0. Below is a gif of two models in rendered viewport. The model on the left has custom normals and the model on the right has painted vertex colors. As you can see both react identical to rotation of light source.

Here's how the shader nodes look like in Blender. I believe most of the magic takes place in the normal map node.

I had a look at the source code for the node and tried to replicate it in godot but my limited knowledge with shaders isn't helping me. Here's the source code for the normal map or so I think. https://github.com/blender/blender/blob/main/source/blender/gpu/shaders/material/gpu_shader_material_normal_map.glsl

Latonya

Answer 5

The issue is very likely the coordinate space in which normals were baked. For start try swapping y and z normal components in the shader.

Frangos

Answer 6

That unfortunately didn't work out.

Here's the code I used for swapping normal components.

shader_type spatial;
void vertex () {
	vec3 normal = COLOR.rgb;
	float y = normal.y;
	normal.y = normal.z;
	normal.z = y;
	NORMAL = normal;
}

Here's how it looks in Godot.

As you can see even without any additional changes, the normal map is off. The model in the right is one with custom normals only and how its is supposed to react to directional lighting. The model on the left is the one with vertex colors.

Latonya

Answer 7

Now it looks like you need to flip the z component too.
Btw you can swizzle vector components in the shading language:

NORMAL = COLOR.xzy;
NORMAL.z = -NORMAL.z;

If this doesn't work exactly, try to play around with different component signs and/or component swapping, and see how it affects the lighting.

Frangos

Answer 8

Okay, Thank you for your help. This didn't exactly work out but I'll see if I can mess with the values. BTW, you mentioned which space they were baked in earlier and I checkd in Blender they are in the Model/Object space.

Latonya

Answer 9

It's hard to give you more specific advice without seeing the models.

Frangos

Answer 10

Would it be possible for us to connect over discord or any private messaging service of your choosing? I could share the .blend files over there. Its just that I have been researching for the past three weeks starting from modelling and custom normals until I hit a roadblock where applying shapekeys in Blender would break the normals. Then I figured out after some looking around that baking the edited custom normals to the model would allow me to change shapekeys without breaking normals and it did work in Blender but I can't get it working in Godot. I looked at the source code for the normal map node in Blender linked above and tried to replicate in Godot but it didn't work probably because the code wasn't right.

Latonya

Answer 11

I don't use discord. The easiest way to resolve this would be to post a minimal godot project with both meshes imported, zip it and attach it to a post here. Someone will likely take a look at it.

Btw, is there a reason you're not using regular normal maps?

Frangos

Answer 12

I couldn't bake the normals to an image texture but that was probably because the model I was trying to bake it from only had the vertex normals edited. I've attached the godot project but removed the shadercache to reduce the size. Also attached is the .blend file.

kusanagi.zip

210kB

godot.zip

7MB

Latonya

Answer 13

xyz Would it be possible for us to connect over discord or any private messaging service of your choosing?

NepNep978 I don't use discord. The easiest way to resolve this would be to post a minimal godot project with both meshes imported, zip it and attach it to a post here.

The forum does support private conversations. Just navigate to the users profile and you should be able to start a private discussion with them there. Drop-down menu towards top-right(for me as an admin anyways). May just be a button with an icon of a sort of folding 3 panels for regular users.

Tiebout

Answer 14

Hm, what I wrote in previous post should ensure the normals on both models have matching orientation, however their orientation will be rotated 90 degrees and both will look off in respect to actual light direction in Godot.

So best way to ensure the correct orientation on both models without re-exporting anything would be to rotate the color-encoded normals by 90 degrees around x axis in the vertex shader:

	const float a = PI/2.0;
	const mat3 mx = mat3(
		vec3(1.0, 0.0, 0.0), 
		vec3(0.0, cos(a), -sin(a)), 
		vec3(0.0, sin(a), cos(a)) 
	);
	NORMAL = mx * (COLOR.xyz * 2.0 - 1.0);

Now both models will have correct normal orientation in respect to light direction in Godot:

However there still seems to be some slight discrepancy. This was probably caused by your transfer/bake process in Blender. So double-check that.

Frangos

Answer 15

This actually worked! I cannot believe all of my woes were for one export setting in Blender.

Regarding the UV Map, I have a seperate blend file with the UVs created but baking the normals would just create a purple image with no deformations.

Anyways, thank you so much for your help!

Latonya

Answer 16

but baking the normals would just create a purple image with no deformations

For this you'd need to bake the normals in object space, instead of default tangent space used for standard normal maps.

Frangos

Answer 17

Okay, I'll look into it.

You've given me a lot of stuff to look into and once again thank you. You've helped me immensely.

Latonya

Answer 18

Just one last thing. Do you know where I can do some reading up on the things you mentioned such as vectors and math in 3D space. I've been looking into the spatial shaders documentation for Godot for the last couple of days but that didn't help me figure out a solution on my own.

Latonya

Answer 19

The key areas of math used in 3d graphics are trigonometry and linear algebra. Linear algebra deals with vectors, matrices, n-dimensional linear spaces etc. It's a very broad field, but only its 2d/3d subset is typically used in practical graphics/game programming.

You can surely find a lot of resources by googling "linear algebra for 3d graphics"

3Blue1Brown has a good series on linear algebra. It may be a bit involved, depending on your affinities and current level of knowledge.
Also there's a decent introductory series at Khan academy

Frangos