Aligning 360° Panoramas with 3D Models in Three.js

When creating a virtual tour that combines 3D models with 360° panoramas, there are several key techniques to ensure proper alignment. Here's how to approach this problem:

Key Considerations

1. Camera Positioning: The camera should be placed at the exact position where the panorama was captured
2. Panorama Orientation: The panorama needs to be correctly oriented to match the 3D model's coordinate system
3. Transition Handling: Smooth transitions between 3D model view and panorama view

Recommended Approach

Instead of manipulating the sphere, a better approach is to:

1. Position the camera at the hotspot location
2. Use the panorama as a background or skybox
3. Handle the transition between views properly

Here's an implementation approach:

JAVASCRIPTCopied
// Create a panorama viewer system
function createPanoramaViewer() {
  // Create a separate scene for the panorama
  const panoScene = new THREE.Scene();
  
  // Create a sphere for the panorama
  const sphereGeometry = new THREE.SphereGeometry(500, 60, 40);
  sphereGeometry.scale(-1, 1, 1); // Invert the sphere so texture is on the inside
  
  // Load the panorama texture
  const textureLoader = new THREE.TextureLoader();
  const panoramaTexture = textureLoader.load('https://i.imgur.com/wJeAbfa.jpeg');
  const panoMaterial = new THREE.MeshBasicMaterial({ map: panoramaTexture });
  
  // Create the panorama sphere
  const panoSphere = new THREE.Mesh(sphereGeometry, panoMaterial);
  panoScene.add(panoSphere);
  
  // Store the original camera position and rotation for returning to 3D view
  let originalCameraPosition = new THREE.Vector3();
  let originalCameraRotation = new THREE.Euler();
  
  // Function to enter panorama view at a specific hotspot
  function enterPanoramaView(hotspotPosition, panoramaRotation = { x: 0, y: 0, z: 0 }) {
    // Store original camera state
    originalCameraPosition.copy(camera.position);
    originalCameraRotation.copy(camera.rotation);
    
    // Move camera to hotspot position
    camera.position.copy(hotspotPosition);
    
    // Apply panorama-specific rotation if needed
    camera.rotation.set(
      panoramaRotation.x || 0,
      panoramaRotation.y || 0,
      panoramaRotation.z || 0
    );
    
    // Switch to rendering the panorama scene
    isInPanoramaMode = true;
  }
  
  // Function to exit panorama view
  function exitPanoramaView() {
    // Restore original camera position and rotation
    camera.position.copy(originalCameraPosition);
    camera.rotation.copy(originalCameraRotation);
    
    // Switch back to rendering the main scene
    isInPanoramaMode = false;
  }
  
  // In your render loop
  function render() {
    if (isInPanoramaMode) {
      renderer.render(panoScene, camera);
    } else {
      renderer.render(mainScene, camera);
    }
    requestAnimationFrame(render);
  }
  
  return {
    enterPanoramaView,
    exitPanoramaView,
    panoScene
  };
}

Handling Orientation

To correctly align the panorama with the 3D model:

1. Metadata Approach: If your panoramas have orientation metadata, use it to set the initial camera rotation
2. Calibration Approach: Create a calibration system where you can adjust and save the orientation for each panorama
3. Reference Points: Use visible reference points in both the 3D model and panorama to align them

JAVASCRIPTCopied
// Example of a hotspot system with orientation data
const hotspots = [
  {
    position: new THREE.Vector3(101.02, 97.17, 4.35),
    panoramaPath: 'https://i.imgur.com/wJeAbfa.jpeg',
    orientation: { x: 0, y: Math.PI/2, z: 0 } // Calibrated orientation
  },
  // More hotspots...
];

// When user activates a hotspot (e.g., by pressing 'm')
function activateHotspot(hotspotIndex) {
  const hotspot = hotspots[hotspotIndex];
  panoramaViewer.enterPanoramaView(hotspot.position, hotspot.orientation);
}

This approach separates the panorama viewing system from your main 3D scene, making it easier to manage transitions and orientation calibration.