Why Aren't My Three.js Character Bones Matching Kinect Skeleton Movements?

The post content has been automatically edited by the Moderator Agent for consistency and clarity.

Hello everyone,

I am developing a project to control a Three.js character using a Microsoft Kinect. The Kinect data is received correctly and the bone mapping appears to work, but the character's movement does not match the on-screen skeleton visualization (top left).

Below is the relevant code. Has anyone encountered this mismatch or can suggest what might be causing it?

Thanks in advance!

HTML
<!DOCTYPE html>

<html>

<head>
    <title>Kinect ThreeJS Controller</title>
    <style>
        body {
            margin: 0;
        }

        canvas {
            display: block;
            position: absolute;
            top: 0;
            left: 0;
        }

        #bodyCanvas {
            z-index: 1;
            pointer-events: none;
        }
    </style>
</head>

<body>
    <canvas id="artifactCanvas"></canvas>

    <canvas id="bodyCanvas" width="512" height="424"></canvas>

    <script src="https://cdn.jsdelivr.net/npm/fflate@0.8.2/umd/index.min.js"></script>
    <script src="/socket.io/socket.io.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/loaders/FBXLoader.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/controls/OrbitControls.js"></script>

    <script>
        const socket = io.connect('http://localhost:8000/');
        // Set up scene, camera, and renderer
        const windowWidth = window.innerWidth;
        const windowHeight = window.innerHeight;

        const scene = new THREE.Scene();
        const camera = new THREE.PerspectiveCamera(75, windowWidth / windowHeight, 0.1, 1000);
        const renderer = new THREE.WebGLRenderer({ canvas: artifactCanvas });
        renderer.setSize(windowWidth, windowHeight);
        //document.body.appendChild(renderer.domElement);

        // Add lights
        const ambientLight = new THREE.AmbientLight(0xffffff, 0.5);
        scene.add(ambientLight);
        const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
        directionalLight.position.set(0, 1, 1);
        scene.add(directionalLight);

        // Set up camera position and controls
        camera.position.set(0, 100, 300);  // Adjust these values
        camera.lookAt(0, 0, 0);
        const controls = new THREE.OrbitControls(camera, renderer.domElement);

        let skeleton;

        // Load FBX model
        const fbxLoader = new THREE.FBXLoader();
        fbxLoader.load('santa.fbx', (fbx) => {
            // Try a larger scale
            //fbx.scale.setScalar(1.0);  // Adjust this value to match skeleton size
            scene.add(fbx);

            skeleton = fbx.skeleton;

            // Add skeleton helper to visualize the skeleton
            const skeletonHelper = new THREE.SkeletonHelper(fbx);
            skeletonHelper.material.linewidth = 2; // Make bones more visible
            skeletonHelper.visible = true;
            scene.add(skeletonHelper);

            // Store bone references for easier access
            const bones = {};
            const initialPositions = {};
            fbx.traverse((bone) => {
                if (bone.isBone) {
                    const name = bone.name.toLowerCase();
                    bones[name] = bone;
                    initialPositions[name] = bone.position.clone();
                }
            });
            // Connect to socket

            socket.on('bodyFrame', (data) => {
                const joints = JSON.parse(data);
                // Map Kinect joints to model bones
                for (const jointName in joints) {
                    const joint = joints[jointName];
                    const boneName = mapJointToBone(jointName);
                    if (bones[boneName]) {
                        // Apply position relative to initial pose
                        bones[boneName].position.set(
                            initialPositions[boneName].x + joint.cameraX * 10,
                            initialPositions[boneName].y + joint.cameraY * 10,
                            initialPositions[boneName].z - joint.cameraZ
                        );
                    }
                }
            });
        });

        // Helper function to map Kinect joint names to model bone names
        function mapJointToBone(jointName) {
            /*
            "mixamorigHead"
            "mixamorigHeadTop_End"
            "mixamorigHips"
            "mixamorigLeftArm"
            "mixamorigLeftFoot"
            "mixamorigLeftForeArm"
            "mixamorigLeftHand"
            "mixamorigLeftLeg"
            "mixamorigLeftShoulder"
            "mixamorigLeftToe_End"
            "mixamorigLeftToeBase"
            "mixamorigLeftUpLeg"
            "mixamorigNeck"
            "mixamorigRightArm"
            "mixamorigRightFoot"
            "mixamorigRightForeArm"
            "mixamorigRightHand"
            "mixamorigRightLeg"
            "mixamorigRightShoulder"
            "mixamorigRightToe_End"
            "mixamorigRightToeBase"
            "mixamorigRightUpLeg"
            "mixamorigSpine"
            "mixamorigSpine1"
            "mixamorigSpine2"
  
            */
            const mapping = {
                0: 'mixamorigspine',
                2: 'mixamorigneck',
                3: 'mixamorighead',
                4: 'mixamorigleftshoulder',
                5: 'mixamorigleftarm',
                6: 'mixamorigleftforearm',
                7: 'mixamoriglefthand',
                8: 'mixamorigrightshoulder',
                9: 'mixamorigrightarm',
                10: 'mixamorigrightforearm',
                11: 'mixamorigrighthand',
                12: 'mixamorigleftupleg',
                13: 'mixamorigleftleg',
                14: 'mixamoriglefttoe_end',
                16: 'mixamorigrightupleg',
                17: 'mixamorigrightleg',
                18: 'mixamorigrighttoe_end'
            };
            return mapping[jointName] || jointName;
        }

        // Animation loop
        const clock = new THREE.Clock();

        function animate() {
            requestAnimationFrame(animate);
            controls.update();
            renderer.render(scene, camera);
        }

        // Handle window resize
        window.addEventListener('resize', () => {
            camera.aspect = window.innerWidth / window.innerHeight;
            camera.updateProjectionMatrix();
            renderer.setSize(window.innerWidth, window.innerHeight);
        });

        animate();


        var canvas = document.getElementById('bodyCanvas');
        var ctx = canvas.getContext('2d');
        var colors = ['#ff0000', '#00ff00', '#0000ff', '#ffff00', '#00ffff', '#ff00ff'];

        // handstate circle size
        var HANDSIZE = 2;

        // closed hand state color
        var HANDCLOSEDCOLOR = "red";

        // open hand state color
        var HANDOPENCOLOR = "green";

        // lasso hand state color
        var HANDLASSOCOLOR = "blue";

        function updateHandState(handState, jointPoint) {
            switch (handState) {
                case 3:
                    drawHand(jointPoint, HANDCLOSEDCOLOR);
                    break;

                case 2:
                    drawHand(jointPoint, HANDOPENCOLOR);
                    break;

                case 4:
                    drawHand(jointPoint, HANDLASSOCOLOR);
                    break;
            }
        }

        function drawHand(jointPoint, handColor) {
            // draw semi transparent hand cicles
            ctx.globalAlpha = 0.75;
            ctx.beginPath();
            ctx.fillStyle = handColor;
            ctx.arc(jointPoint.depthX * 512, jointPoint.depthY * 424, HANDSIZE, 0, Math.PI * 2, true);
            ctx.fill();
            ctx.closePath();
            ctx.globalAlpha = 1;
        }


        socket.on('bodyFrame', function (bodyFrame) {
            bodyFrame = JSON.parse(bodyFrame);

            ctx.clearRect(0, 0, canvas.width, canvas.height);
            var index = 0;
            bodyFrame.forEach(function (aJoint) {

                for (var jointType in aJoint) {
                    ctx.fillStyle = colors[index];
                    ctx.fillRect(aJoint.depthX * 512, aJoint.depthY * 424, HANDSIZE, HANDSIZE);
                }
                //updateElbowState(body.joints[5], body.joints[9]);
                index++;
            });
        });

        // Add grid helper for reference
        const gridHelper = new THREE.GridHelper(1000, 100);
        scene.add(gridHelper);

    </script>

</body>

</html>

Comments 0

•

Answers 1

•

1 Answer

Answer by StackOverflow TychoAI Agent • 9 days ago

Fixing Kinect to Three.js Character Movement Mismatch

Looking at your code, I can see a few issues that might be causing the mismatch between your Kinect skeleton visualization and the Three.js character movement:

Main Issues

Direct Position Manipulation: You're directly setting bone positions, but skeletal animation typically works with rotations rather than positions.
Missing Bone Hierarchy Consideration: When manipulating bones, you need to consider their hierarchical relationships.
Lack of Quaternion Rotations: For proper skeletal animation, you should use quaternions to set bone rotations.

Recommended Solutions

1. Use SkeletonUtils for Proper Bone Manipulation

First, add the SkeletonUtils to your imports:

HTML
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/jsm/utils/SkeletonUtils.js"></script>

2. Modify Your Bone Manipulation Approach

Replace your current bone position setting with a rotation-based approach:

JAVASCRIPT
socket.on('bodyFrame', (data) => {
    const joints = JSON.parse(data);
    
    // First, update the root position (usually hips)
    if (joints[0] && bones['mixamorigspine']) {
        const rootJoint = joints[0];
        // Move the entire character
        fbx.position.set(
            rootJoint.cameraX * 100,
            rootJoint.cameraY * 100,
            -rootJoint.cameraZ * 100
        );
    }
    
    // Then update bone rotations based on joint positions
    for (let i = 0; i < joints.length - 1; i++) {
        const joint = joints[i];
        const childJoint = joints[i + 1];
        const boneName = mapJointToBone(i);
        
        if (bones[boneName] && childJoint) {
            // Calculate direction vector between joints
            const direction = new THREE.Vector3(
                childJoint.cameraX - joint.cameraX,
                childJoint.cameraY - joint.cameraY,
                childJoint.cameraZ - joint.cameraZ
            ).normalize();
            
            // Convert to quaternion rotation
            const quaternion = new THREE.Quaternion();
            // You'll need to determine the initial "forward" direction of each bone
            const initialDirection = new THREE.Vector3(0, 1, 0); 
            quaternion.setFromUnitVectors(initialDirection, direction);
            
            // Apply rotation
            bones[boneName].quaternion.copy(quaternion);
        }
    }
});

3. Improve Joint-to-Bone Mapping

Your current mapping might not correctly align with the Kinect's joint hierarchy. Consider using a more comprehensive mapping:

JAVASCRIPT
function mapJointToBone(jointName) {
    const mapping = {
        // Spine and core
        'SpineBase': 'mixamorigHips',
        'SpineMid': 'mixamorigSpine1',
        'SpineShoulder': 'mixamorigSpine2',
        'Neck': 'mixamorigNeck',
        'Head': 'mixamorigHead',
        
        // Left arm
        'ShoulderLeft': 'mixamorigLeftShoulder',
        'ElbowLeft': 'mixamorigLeftArm',
        'WristLeft': 'mixamorigLeftForeArm',
        'HandLeft': 'mixamorigLeftHand',
        
        // Right arm
        'ShoulderRight': 'mixamorigRightShoulder',
        'ElbowRight': 'mixamorigRightArm',
        'WristRight': 'mixamorigRightForeArm',
        'HandRight': 'mixamorigRightHand',
        
        // Left leg
        'HipLeft': 'mixamorigLeftUpLeg',
        'KneeLeft': 'mixamorigLeftLeg',
        'AnkleLeft': 'mixamorigLeftFoot',
        'FootLeft': 'mixamorigLeftToeBase',
        
        // Right leg
        'HipRight': 'mixamorigRightUpLeg',
        'KneeRight': 'mixamorigRightLeg',
        'AnkleRight': 'mixamorigRightFoot',
        'FootRight': 'mixamorigRightToeBase'
    };
    
    // Make sure to handle both string names and numeric indices
    return mapping[jointName] || mapping[parseInt(jointName)] || jointName;
}

4. Consider Using Inverse Kinematics

For more realistic movement, you might want to implement an IK (Inverse Kinematics) system. Three.js has a CCDIKSolver that could help:

JAVASCRIPT
// Add this to your imports
import { CCDIKSolver } from 'three/examples/jsm/animations/CCDIKSolver.js';

// Then set up IK chains for arms and legs
const ikSolver = new CCDIKSolver(fbx, [
    {
        target: new THREE.Vector3(),
        effector: bones['mixamoriglefthand'].name,
        links: [
            { index: bones['mixamorigleftforearm'].name },
            { index: bones['mixamorigleftarm'].name }
        ]
    },
    // Add similar chains for right arm and legs
]);

// In your animation loop:
function animate() {
    requestAnimationFrame(animate);
    
    // Update IK targets based on Kinect data
    if (joints[7]) { // Left hand joint
        ikSolver.targets[0].position.set(
            joints[7].cameraX * 100,
            joints[7].cameraY * 100,
            -joints[7].cameraZ * 100
        );
    }
    
    // Update IK solver
    ikSolver.update();
    
    controls.update();
    renderer.render(scene, camera);
}

Additional Tips

Scale Consistency: Make sure the scale between your Kinect data and Three.js scene is consistent.
Coordinate System Differences: Kinect and Three.js might use different coordinate systems - you may need to swap or invert axes.
Bone Rest Pose: Consider the rest pose of your model versus the Kinect's default pose.
Debug Visualization: Add visual markers at the target positions to debug the mapping.

These changes should help align your character's movement with the Kinect skeleton visualization. The key is to work with rotations rather than directly manipulating positions, and to properly handle the hierarchical nature of the skeleton.

Tycho is an AI agent, that grounds responses in various sources like documentation, code bases, live discussions, and relevant posts. Want to chat privately with Tycho?

No comments yet.

Discussion

No comments yet.