Removing Wall/Floor Surfaces from Images with GLSL

Looking at your problem, I can see you're trying to achieve a transparent effect for floor/wall surfaces while keeping objects intact. This is a challenging computer vision task that's difficult to solve with simple GLSL alone.

After reviewing your code and examples, here are some approaches that might help:

1. Color-Based Segmentation with GLSL

Your current approach seems to be using color thresholds. This can be improved by:

GLSLCopied
uniform sampler2D u_texture;
varying vec2 v_uv;

void main() {
  vec4 color = texture2D(u_texture, v_uv);
  
  // Detect floor/wall by color range (adjust these values)
  float floorness = smoothstep(0.7, 0.9, color.r) * 
                    smoothstep(0.7, 0.9, color.g) * 
                    smoothstep(0.7, 0.9, color.b);
  
  // Apply transparency based on "floorness"
  color.a = 1.0 - floorness;
  
  gl_FragColor = color;
}

2. Edge Detection + Color Segmentation

Combine edge detection with color segmentation for better results:

GLSLCopied
uniform sampler2D u_texture;
uniform vec2 u_resolution;
varying vec2 v_uv;

void main() {
  vec2 pixelSize = 1.0 / u_resolution;
  vec4 center = texture2D(u_texture, v_uv);
  
  // Simple Sobel edge detection
  vec4 n = texture2D(u_texture, v_uv + vec2(0.0, -pixelSize.y));
  vec4 e = texture2D(u_texture, v_uv + vec2(pixelSize.x, 0.0));
  vec4 s = texture2D(u_texture, v_uv + vec2(0.0, pixelSize.y));
  vec4 w = texture2D(u_texture, v_uv + vec2(-pixelSize.x, 0.0));
  
  float edgeness = length((n - s).rgb) + length((e - w).rgb);
  edgeness = smoothstep(0.1, 0.3, edgeness);
  
  // Color-based floor detection (adjust for your images)
  float floorness = smoothstep(0.6, 0.8, center.r) * 
                    smoothstep(0.6, 0.8, center.g) * 
                    smoothstep(0.6, 0.8, center.b);
  
  // Keep edges and non-floor colors
  float alpha = max(edgeness, 1.0 - floorness);
  
  gl_FragColor = vec4(center.rgb, alpha);
}

3. JavaScript Pre-processing

For more complex segmentation, consider pre-processing with JavaScript:

JAVASCRIPTCopied
function preprocessImage(imageData) {
  const data = imageData.data;
  
  for (let i = 0; i < data.length; i += 4) {
    const r = data[i];
    const g = data[i + 1];
    const b = data[i + 2];
    
    // Detect floor/wall colors (adjust thresholds)
    if (r > 180 && g > 180 && b > 180 && 
        Math.abs(r - g) < 20 && Math.abs(r - b) < 20) {
      data[i + 3] = 0; // Make transparent
    }
  }
  
  return imageData;
}

Recommendations

For your specific case, I recommend:

1. Combine approaches: Use JavaScript for initial segmentation, then refine with GLSL
2. Use HSL color space: Convert RGB to HSL in your shader for better color segmentation
3. Add user controls: Let users adjust thresholds for different images

For complex images like yours, a pure GLSL solution might not be sufficient. You might need to consider:

• Using a pre-trained ML model (via TensorFlow.js)
• Implementing more sophisticated segmentation algorithms
• Using WebAssembly for more complex image processing

The most effective approach would likely combine multiple techniques rather than relying on a single method.