Real Time Motion Capture Using a Single Time-Of-Flight Camera Varun Ganapathi, Christian Plagemann, Daphne Koller, Sebastian Thrun CVPR 2010 Q36981123 邱碁森

Outline Introduction Probabilistic Model Inference Experiments Conclusions

Introduction Motion capture is used to human-machine interaction, smart surveillance and so on. Time-of-flight sensors offers rich sensory information, not sensitive to changes in lighting, shadows, and some other problems. This paper propose an efficient filtering algorithm for tracking human pose for fast operation at video frame.

What is Probabilistic Model? A tree-shaped kinematic chain (skeleton) Human body is modeled as 15 body parts The transformations of the body Xt at time t is a set: Xt= {Xi}, i = 1~15 X1: the root of tree → the pelvis part root(pelvis): could freely rotate and translate other parts: connected to the their parent, allow to rotate (not to translate)

What is Probabilistic Model? (cont.) The absolute orientation of a body part i: Wi(X) multiplying the transformations of its ancestors in the kinematic chain Wi(X) = X1∙X2 · ...· Xparent(i) ∙ Xi

Why need the Probabilistic Model? Determine the most likely state at at time t the pose set Xt the first discrete-time derivative set Vt (velocities) zt: the recorded range measurements The system is modeled as a dynamic Bayesian network (DBN)

Probabilistic Model The measured range scan is denoted by z = {zk} k=1M where zk gives the measured depth of the pixel at coordinate k.

Probabilistic Model Assumption: the accelerations in our system are drawn from a Gaussian distribution with zero mean

Inference How to perform efficient inference at each frame? Model Based Hill Climbing Search (HC) A component locally optimizes the likelihood function Evidence Propagation (EP) An inference procedure generate likely states which are used to initialize the HC Inference n. 推論機

Model Based Hill Climbing Search coarse-to-fine The procedure can then potentially be applied to a smaller interval about the value chosen at the coarser level hill-climbing Start from the base of kinematic chain which includes the largest body parts, and proceed toward the limbs 3 2 sample: 0.5 0.45 0.4 ... -0.35 -0.4 -0.45 -0.5 1 then chose the best one optimize the X axis

Evidence Propagation Problem: fast motion cause motion blur occlusion cause the estimate of the state of hidden parts to drift the likelihood function has ridges (difficult to navigate) This procedure that identifies promising locations for body parts to find likely poses

Evidence Propagation Steps in this procedure: Body Part Detection: identify possible body part locations from the current range image Probabilistic Inverse Kinematics: update the body configuration X given possible correspondences between mesh vertices and part detections Data Association and Inference: determine the best subset of such correspondences

Body Part Detection Five body parts: head, left hand, right hand, left foot and right foot are found from the current range image. Interest Point(AGEX) Detection start on the geodesic centroid of the mesh: AGEX1(M) recursively find the vertex AGEXk(M) which has max geodesic distance to AGEXk-1(M) Identification of Parts points are classified as body part by training these data using a marker-based motion capture system( LED mark) C. Plagemann, V. Ganapathi, D. Koller, and S. Thrun. Realtime identification and localization of body parts from depth images. In IEEE Int. Conference on Robotics and Automation (ICRA), Anchorage, Alaska, USA, 2010.

Evidence Propagation

Experiments Using a Swissranger SR4000 Time-of-Flight camera Tracking results on real-world test sequences, sorted from most complex (left) to least complex (right).

Experiments A Tennis sequence Only use Model-Based search Our combined tracker

Conclusions A novel algorithm for combining part detections with local hill-climbing for marker less tracking of human pose. With the hybrid, GPU-accelerated filtering approach