2. 3d Pose Detection Used by Kinect
Accurate when the pose closely matches a stored pose
Inaccurate when novel poses are made
Can often produce shaky movement due to pose snapping

3. 3d Pose Tracking
Calculate poses based on previous poses and current data
No datasets required
Has issues with local minima

4. Hybrid Store a collection of preprocessed poses
Start session with a detected pose
Track until a failure is hit
Detect that frame, and continue tracking from there

5. Pose Tracking in Detail
Given the current set of pixels, and the calculated poses of the past few frames, calculate the most probable current pose
Repeat the following until error is reduced
Calculate a hypothesized 3d representation (rendered image) using the 3d camera image (observed data)
Calculate quite a lot about the rendered image:
Edepth, Eextra, Esilhouette, Eprior
Adjust the hypothesized pose accordingly
If that fails significantly, “detect” the current pose and try again

6. Optimizing the Position
Solve energy minimization problems:
Edepth: Depth data, excluding any rendered foreground pixels that match background observed pixels
Eextra: The above depth term, but with added calculations where rendered pixels don’t match with observed pixels
Esilhouette: The difference in silhouettes (pixel is in foreground of one image but the background of the other)
Eprior: Evaluation of the rendered pose based on the previous rendered poses

7. More Pose Detecting
Grab a random subset of pixels and match them to known poses
Throw out uncertain data
20k poses total

8. Setup User must stand in a specific pose for calibration
Select random subset of pixels to determine widths of limbs

9. Results

10. Results Roughly 1 in 200 frames reinitialized by the detector
Very fast movement often causes local minima
On average, 1 failure every 33 seconds
High performance running in parallel on the GPU

12. Further Work Use skinned mesh models Increase pose set
Use color data in the algorithm as well