1. Recovering Articulated Object Models from 3D Range Data Dragomir Anguelov Daphne Koller Hoi-Cheung Pang Praveen Srinivasan Sebastian Thrun Computer Science Dept. Stanford University

2. Problem overview Data: Articulated object observed in different configurations 3D range surfaces (meshes) obtained by a scanner Task: Recover the articulated object model 1. Register all surfaces 2. Cluster the object surface into rigid parts 3. Recover joints linking the parts Applications of articulated models Animation, object detection, scene segmentation

3. Surface registration Algorithm input: meshes Z 0, Z 1, … Z N Pick template mesh X = Z 0 Need to recover point-to-point correspondences between X and instance meshes Z i, i={1, …, N} Any registration algorithm providing can be used Correlated Correspondences algorithm [Anguelov et al ‘04] Unsupervised nonrigid registration algorithm Handles large deformations, occlusion (but not clutter) Algorithm Overview Computes embedding of mesh Z into mesh X defined as a joint assignment to correspondences Encodes correspondence correlations (minimizing deformation, preserving geodesic distances) as pairwise potentials in a Markov network over Uses Loopy-BP to obtain a good joint assignment to all variables 22 11 33

4. Clustering into rigid parts: Generative model Template X Transformed Model Instance Z Transformations z1z1 b1b1 a1a1 Tx1x1 y1y1 aMaM xMxM yMyM zkzk bkbk … … … … Part labels Points Point labels Points  j = Part (x j )  k = ModelPoint (z k ) Assume  values provided by a nonrigid registration algorithm 1. Rigid part points share the same rigid transformaition in each instance rotationtranslation 2. Each part is a connected region of the template mesh surface a) Soft contiguity constraints: If and are connected in X for b) Hard contiguity constraints: Each part consists of a single connected component of the surface of X

5. Clustering into rigid parts: Optimization Model Summary: Ignoring hard contiguity constraints, generative model defines a Markov Network over the part labels : Objective Iterative Optimization E-step: Given transformations, compute the part labels maximizing Eq. (1) and satisfying the hard contiguity constraints M-step: Given part labels, compute transformations Point in mesh Z i, generated by template point x j

6. Clustering into rigid parts: E-Step The problem of finding a MAP assignment to given which maximizes Eq. (1) can be expressed as an integer program (IP). A relaxation of the IP produces a linear program (LP) [Kleinberg et al ’99]. For 2-part models, the optimal LP solution is also the optimal IP solution. For n-part models, n>2: In case of a fractional LP solution, a rounding procedure produces an IP solution within a factor of 2 of the optimal. The LP can be solved very efficiently by using multiway min-cuts [Kolmogorov et al ’04] The LP solution may not satisfy hard contiguity constraints. A part consisting of m connected components is split into m separate parts which doesn’t change the objective in Eq. (1).

7. Clustering into rigid parts: LP formulation Introduce variables s.t. Log-cost associated with potential becomes where Define separation cost of an edge as where Putting everything together: Relax binary constraints to obtain LP

8. Clustering into rigid parts: M-step and Annealing M-step: find rigid part transformations given part labels Optimization function decomposes into a separate equation for each : Eq. (2) is isomorphic to the ICP problem [Besl’ 92]. Close form estimate of is obtained by solving a simple LP. Annealing Number of parts in solution depends on the tradeoff where large values of result in fewer parts When estimate of is poor and is large, solving E-step LP is hard, because multiple plausible solutions with similar scores exist Starting with low values of and gradually increasing performs best in practice

9. Clustering into Rigid Parts: Initialization Need good initial values for the transformations Template X Method 1: Cluster local rigid transformations [Cheung et al ’03] Associate a vector of transformations with each point (estimate using the known correspondences for all instance meshes) Cluster points into k parts using adaptive PCA [Archer et al ’02] Slow: >10 mins for 2000 point models Method 2: Break surface into k arbitrary fragments of approximately equal size Fast: <1s for 2000 point models Same final results! a) Rigid parts after initialization b) Rigid parts after EM a) b)

10. Optimizing the Joint Positions A joint is a point belonging to two parts simultaneously: [Cheung et al ’03] To solve for the joint, must optimize Sometimes, the solution can be an entire space of points Example: a joint with one degree of freedom Introduce an additional regularization term, enforcing that the joint is close to the boundary between two parts in X a) b) a) Rigid parts after EM b) Estimated joints Joint between parts p and q

11. Results: Puppet Dataset Correctly identifies all 15 puppet parts and the skeleton

12. Results: Initialization Testing EM Initialization Initialize with a different number of parts k If k > 15, the algorithm consistently finds the correct number of parts (15)

13. Results: Arm Dataset Correctly identifies 4 arm parts and the skeleton in the presence of non-rigid deformation

14. Results: Annealing Results of annealing with increasing value of Increasing decreases the number of parts (A – D) Increasing too much results in a partition which tries to reduce the number of links between parts (E)

15. Conclusions and Future Work Algorithm for recovering articulate object models from 3D data Clusters the object surface into rigid parts Recovers the articulated object skeleton Algorithm properties Probabilistic formulation Efficient global inference Algorithm recovers complex models consisting of a large number of parts in real world datasets Future work Extend the algorithm to allow and explicitly model (small) deformations of the articulated parts Robustness to local maxima