Enforcing Constraints for Human Body Tracking David Demirdjian Artificial Intelligence Laboratory, MIT

WOMOT 2003 Goal Real-time articulated body tracking from stereo accounting for constraints on pose

WOMOT 2003 Approach Differential tracking: assuming the articulated body pose  t-1 is known, estimate the pose  t (or equivalently the set of limb rigid motions  i =(t i  i ) between poses  t-1 and  t ) that minimizes the distance between the articulated model and the observed 3D data  tracking as a constrained optimization problem

WOMOT 2003 Approach Differential tracking: assuming the articulated body pose  t-1 is known, estimate the pose  t (or equivalently the set of limb rigid motions  i =(t i  i ) between poses  t-1 and  t ) that minimizes the distance between the articulated model and the observed 3D data  tracking as a constrained optimization problem –Solve unconstrained optimization problem –Project solution on constraint surface

WOMOT 2003 Projection-based approach   unconstrained optimum)  human motion manifold

WOMOT 2003 Approach Estimate limb motions  i =(t i  i ) independently using standard multi-object tracking algorithm Projection: find the closest body motion   =(  i ’) to  =(  i ) that satisfies human body constraints: –articulated constraints –other constraints: joint limit, …

WOMOT 2003 Previous work Particle sampling: Sidenbladh & al. ECCV’00 Sminchisescu & Triggs CVPR’01 Differential tracking: Plankers & Fua ICCV’99 Jojic & al. ICCV’99 Delamarre & Faugeras ICCV’99

WOMOT 2003 Plan Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion

WOMOT 2003 Multi-object tracking Assuming the articulated body pose  t-1 is known, estimate the set of limb rigid motions  i =(t i  i ) minimizes the distance between the (moved) limb and the observed 3D data Consists in estimating limb motions  i =(t i  i ) independently: –Estimate visible 3D mesh of each limb –Current implementation uses the ICP algorithm to register each limb w.r.t 3D data

WOMOT 2003 Iterative Closest Point 3D registration –find the rigid transformation  that maps shape S t (limb model) to shape S r (3D data) SrSr StSt 

WOMOT 2003 Iterative Closest Point Matching points For all points in S t, we search for the closest point in S r by computing the distance and keep the closest SrSr StSt

WOMOT 2003 Iterative Closest Point Energy function minimization Estimate the rigid transformation that minimizes the sum of squared distances between corresponding matched points SrSr StSt

WOMOT 2003 Iterative Closest Point Energy function minimization Estimate the rigid transformation that minimizes the sum of squared distances between corresponding matched points SrSr StSt

WOMOT 2003 Iterative Closest Point Optimal rigid transformation  (and uncertainty   ) found by combining all the elementary displacements

WOMOT 2003 Plan Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion

WOMOT 2003 Projection The unconstrained optimal body motion is given by  =(  1,  2 …  N ) With uncertainty  =(  1,  2 …  N ) with  =(  1 ’,  2 ’ …  N ’) satisfying articulated constraints Articulated constraints enforcement: find  that minimizes the Mahalanobis distance:

WOMOT 2003 Articulated motion estimation If M ij is a joint between objects i and j: M ij joint (R i,t i ) (R j,t j ) obj. i obj. j Motion of point M ij on limb i Motion of point M ij on limb j =

WOMOT 2003 Articulated motion estimation If M ij is a joint between objects i and j: M ij joint (R i,t i ) (R j,t j ) obj. i obj. j Motion of point M ij on limb i Motion of point M ij on limb j =

WOMOT 2003 Articulated motion estimation If M ij is a joint between objects i and j: M ij joint (R i,t i ) (R j,t j ) obj. i obj. j Motion of point M ij on limb i Motion of point M ij on limb j = [.] x denotes skew-symmetric matrix

WOMOT 2003 Articulated motion estimation If M ij is a joint between objects i and j: M ij joint (R i,t i ) (R j,t j ) obj. i obj. j Motion of point M ij on limb i Motion of point M ij on limb j = [.] x denotes skew-symmetric matrix

WOMOT 2003 Articulated motion estimation If M ij is a joint between objects i and j: M ij joint (R i,t i ) (R j,t j ) obj. i obj. j Motion of point M ij on limb i Motion of point M ij on limb j =  =(  1 ’,  2 ’ …  N ’)

WOMOT 2003 Articulated motion estimation If M ij is a joint between objects i and j: M ij joint (R i,t i ) (R j,t j ) obj. i obj. j Motion of point M ij on limb i Motion of point M ij on limb j = (Stack for all joints)  =(  1 ’,  2 ’ …  N ’)

WOMOT 2003 Articulated motion estimation All the joint constraints can be written as a linear constraint:  is a linear combination of vectors in the nullspace of  Therefore there exists a matrix V such that: V can be estimated by SVD of 

WOMOT 2003 Articulated motion estimation unconstrained motion articulated motion Find minimum of E 2 in nullspace of … (linear projection)

WOMOT 2003 Plan Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion

WOMOT 2003 Other constraints Constraints: –Static: Joint angle bounds, gravity law, … –Dynamic: Maximum velocity, … Motivation: –Using more constraints to reduce local minima and therefore increase tracking robustness –Application context can reduce tremendously the dimension of the pose space

WOMOT 2003 Other constraints Pose constraints modeled by a (user-defined) function f, such that valid poses correspond to f(  )>0 ex: f(  )=min(g 1 (  ), g 2 (  ), … g N (  )) withg 1 (  ) = angle(l-arm, l-forearm) – min_angle g 2 (  ) = max_angle - angle(l-arm, l-forearm) …. Constraints enforcement: find  * that minimizes the Mahalanobis distance: with  * satisfying F  t-1 (  *)=f(  *(  t-1 ))>0

WOMOT 2003 Other constraints articulated motion articulated constrained motion with (local parameterization)

WOMOT 2003 Constrained optimization algorithm Alternate between binary and stochastic searches 

WOMOT 2003 Constrained optimization algorithm Alternate between binary and stochastic searches 

WOMOT 2003 Constrained optimization algorithm Alternate between binary and stochastic searches  E 2 = E0

WOMOT 2003 Constrained optimization algorithm Alternate between binary and stochastic searches  E 2 = E1

WOMOT 2003 Constrained optimization algorithm Alternate between binary and stochastic searches  

WOMOT 2003 TRACKING SEQUENCE

WOMOT 2003 Future work Quantitative measurement (comparing results with tethered motion capture system) Appearance/Shape information (learning color distribution + shape of limbs) Motion/gesture (including dynamic constraints) Learning human motion constraints (instead of giving them explicitly.. [ICCV’03])

WOMOT 2003 Applications Multimodal Human-Computer Interaction (gesture + speech)

WOMOT 2003

Conclusion Projection-based approach for articulated body tracking –articulated constraints enforced by (linearly) projecting unconstrained limb motion on articulated motion manifold –other constraints enforced using a stochastic constrained optimization algorithm