Vertex Deformation and Normal Recalculation in Three.js

When you're deforming vertices in a shader, you need to recalculate the normals to ensure proper lighting and shadows. The flat appearance you're seeing is because the normals aren't being updated to match your deformed geometry.

The Issue

When you modify vertex positions with:

GLSLCopied
position.y.addAssign(heightNode2);

The normals remain unchanged, which causes incorrect lighting calculations. This is why your terrain appears flat despite having height variations.

Solution Approaches

For vertex shader deformations, you have a few options:

1. Recalculate normals in the shader: You'll need to compute new normals based on neighboring vertices in your shader.

2. Use a normal map: If your height deformation is predictable, you could pre-compute a normal map.

3. Compute derivatives: For heightmap-based terrain, you can compute normals from the height gradient.

Here's a simplified example of how you might recalculate normals in TSL:

GLSLCopied
// Get neighboring heights
float heightL = texture2D(heightMap, uv - vec2(texelSize.x, 0.0)).r;
float heightR = texture2D(heightMap, uv + vec2(texelSize.x, 0.0)).r;
float heightD = texture2D(heightMap, uv - vec2(0.0, texelSize.y)).r;
float heightU = texture2D(heightMap, uv + vec2(0.0, texelSize.y)).r;

// Calculate new normal using central differences
vec3 newNormal = normalize(vec3(
    heightL - heightR,
    2.0, // Adjust this scale factor based on your terrain
    heightD - heightU
));

// Apply the new normal
normal = newNormal;

For your specific case with a buffer attribute, you might need to sample neighboring values from your buffer and compute the gradient that way.

Shadows Consideration

For proper shadows with deformed geometry, ensure:

1. Your light's shadow settings are properly configured
2. The shadow camera's frustum covers your scene
3. The shadow map resolution is adequate

Remember that shadows in Three.js are handled by classes like DirectionalLightShadow and SpotLightShadow, which need proper configuration to work with dynamically deformed geometry.

I recommend checking your JSFiddle and implementing normal recalculation as the first step to fixing your lighting issues.

Fixing Normals After Vertex Deformation in TSL Shader

When you deform vertices in a shader (like you're doing with position.y.addAssign(heightNode2)), the normals don't automatically update to match the new geometry. This is why your shading appears flat - the normals are still pointing in their original directions, which no longer match the deformed surface.

The Problem

Your issue is related to shadows and lighting because:

1. The normals of your mesh aren't being recalculated after vertex deformation
2. Lighting calculations (including shadows) depend on accurate normal vectors
3. When using vertex displacement in shaders, you need to manually update the normals

Solution Approaches

When modifying vertex positions in a shader, you need to recalculate the normals. Here are a few approaches:

1. Calculate Normals in the Shader

You can compute new normals directly in your TSL shader by:

GLSLCopied
// After deforming position
// Calculate new normals based on neighboring vertices
// This is a simplified example - you'll need to adapt to your specific case
vec3 newNormal = normalize(cross(
    dFdx(position.xyz),
    dFdy(position.xyz)
));
normal = normalMatrix * newNormal;

2. Use Normal Map

If your height displacement follows a predictable pattern, you could pre-compute a normal map that corresponds to your height map and use that instead.

3. Compute Derivatives

For terrain-like deformations, you can compute the normal based on the gradient of your height field:

GLSLCopied
// If heightNode2 is your height value
// You need to compute the gradient (how height changes in x and z directions)
float heightL = /* height value to the left */;
float heightR = /* height value to the right */;
float heightU = /* height value up */;
float heightD = /* height value down */;

vec3 newNormal = normalize(vec3(
    heightL - heightR,
    2.0, // Scale factor for normal steepness
    heightD - heightU
));

normal = normalMatrix * newNormal;

Specific to Three.js Shadows

Since you're specifically concerned about shadows, make sure:

1. Your light's shadow settings are properly configured
2. The shadow camera's frustum covers your scene
3. Shadow bias is set appropriately to avoid shadow acne

For example, with DirectionalLightShadow or SpotLightShadow, you might need to adjust the shadow camera parameters.

I recommend checking out the Three.js documentation on LightShadow and its subclasses for more specific shadow-related settings that might help improve your results after fixing the normals issue.