Improving Anti-Aliasing in Grid Pattern Shader

The jagged edges in your grid pattern are occurring because you're using the step() function, which creates a hard edge with no anti-aliasing. To achieve smoother edges, you should replace the step() function with smoothstep() which provides a gradual transition between values.

Here's how you can modify your shader to achieve smoother edges:

GLSLCopied
uniform vec2 uSize; // The size of the plane in meters
uniform float uSquareSize; // Square size in meters
uniform float uGapSize; // Gap size in meters
uniform float uInvert;

varying vec2 vUv;

void main() {
  // Calculate total size of one grid cell (square + gap)
  float gridCellSizeX = uSquareSize + uGapSize;
  float gridCellSizeY = uSquareSize + uGapSize;

  // Calculate number of complete grid cells that fit in each dimension
  float numberOfColumns = floor(uSize.x / gridCellSizeX);
  float numberOfRows = floor(uSize.y / gridCellSizeY);

  // Calculate total width and height occupied by these grid cells
  float totalGridWidth = numberOfColumns * gridCellSizeX;
  float totalGridHeight = numberOfRows * gridCellSizeY;

  // Calculate remaining space after fitting complete grid cells
  float restX = uSize.x - totalGridWidth;
  float restY = uSize.y - totalGridHeight;

  // Calculate offsets to center the pattern
  float offsetX = (restX - uGapSize) / 2.0;
  float offsetY = (restY - uGapSize) / 2.0;

  // Adjust UV coordinates to apply centering offsets
  vec2 adjustedUV = vUv * uSize - vec2(offsetX, offsetY);

  // Calculate step value for square and gap transitions
  float stepValueX = uGapSize / gridCellSizeX;
  float stepValueY = uGapSize / gridCellSizeY;

  // Calculate the modulo values
  float modX = mod(adjustedUV.x / gridCellSizeX, 1.0);
  float modY = mod(adjustedUV.y / gridCellSizeY, 1.0);

  // Define smoothing factor - adjust this value to control the amount of anti-aliasing
  float smoothingFactor = 0.01;

  // Use smoothstep instead of step for smoother transitions
  float patternX = smoothstep(stepValueX - smoothingFactor, stepValueX + smoothingFactor, modX);
  float patternY = smoothstep(stepValueY - smoothingFactor, stepValueY + smoothingFactor, modY);

  // Combine patterns for both dimensions
  float pattern = patternX * patternY;

  // Output color based on pattern
  pattern = mix(pattern, 1.0 - pattern, uInvert);

  // Keep result between 0 and 0.95
  pattern = clamp(pattern, 0.0, 0.95);

  gl_FragColor = vec4(vec3(0.9), pattern);
}

Key changes:

1. Replaced step() with smoothstep() which creates a gradual transition
2. Added a smoothingFactor to control the width of the anti-aliased edge
3. Extracted the modulo calculations to separate variables for clarity

Additionally, you might want to consider:

1. Adjusting the smoothingFactor based on the pixel density or viewing distance

2. Making sure your renderer has anti-aliasing enabled:

   JAVASCRIPTCopied
   const renderer = new THREE.WebGLRenderer({ 
     antialias: true 
   });

3. Setting an appropriate pixel ratio:

   JAVASCRIPTCopied
   renderer.setPixelRatio(window.devicePixelRatio);

The smoothstep() function will create a gradual transition at the edges of your grid squares, which should significantly reduce the jagged appearance.