Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge

Roadmap Brief Introductions Our Framework Experimental Results Summary

Motivation + 3D Shapes Uncertainty Measurements 2D Images Tasks: –To recover deformable shapes from a single image with arbitrary camera viewpoint.

Previous Work Rigid shapes [Prasad’05, Rother’09, Yu’09, etc.] Problems: –Cannot handle self-deformation or articulations. Category-specific articulated shapes e.g., human bodies [Anguelov’05, Balan’07, etc.] Problems: –Requiring strong shape or anatomical knowledge of the category, such as skeletons and joint angles. –Too many parameters to estimate; –Hard to be generalised to other object categories.

Roadmap Brief Introductions Our Framework Experimental Results Summary

Our Contribution A probabilistic framework for: –Modelling different shape variations of general categories; –Synthesizing new shapes of the category from limited training data; –Inferring dense 3D shapes of deformable or articulated objects from a single silhouette;

Explanations on the Graphical Model Shape SynthesisMatching Silhouettes Pose Generator Shape Generator Joint Distribution:

Generating Shapes Target: Simultaneous modelling two types of shape variations: –Phenotype variation: fat vs. thin, tall vs. Short... –Pose variation: articulation, self deformation,... Training two GPLVMs: –Shape generator (M S ) for phenotype variation; –Pose generator (M A ) for pose variation.

Shape Generator (M S ) –Training Set: Shapes in the canonical pose. –Pre-processing: Automatically register each instance with a common 3D template; 3D shape context matching and thin-plate spline interpolation; Perform PCA on all registered 3D shapes. –Input: PCA coefficients of all the data. Generating Shapes

Pose Generator (M A ) –Training Set: Synthetic 3D poses sequences. –Pre-processing: Perform PCA on both spatial positions of vertices and all vertex-wise Jacobian matrices. –Input: PCA coefficients of all the data

Shape Generator (M S ) –Training Set: shapes in the canonical pose. –Input: PCA coefficients of vertex positions. Generating Shapes

Pose Generator (M A ) –Training Set: Synthetic 3D poses sequence. –Input: PCA coefficients of vertex positions and vertex-wise Jacobian matrices.

Shape Synthesis Zero Shape V 0 Pose Generator M A Shape Generator M S VAVA VAVA VSVS VSVS Shape Synthesis V V

Shape Synthesis Modelling the local shape transfer –Computing Jacobian matrices on the zero shape vertex-wisely. JiJi

Shape Synthesis Synthesizing fully-varied shape V from phenotype-varied shape V S and pose- varied shape V A. Probabilistic formulation: a Gaussian Approximation

Matching Silhouettes A two-stage process: o Projecting the 3D shape onto the image plane o Chamfer matching of silhouettes Maximizing likelihood over latent coordinates x A, x S and camera parameters γ k o Optimizing the closed-form lower bound. o Adaptive line-search with multiple initialisations.

Roadmap Brief Introductions Our Framework Experimental Results Summary

Experiments on Shape Synthesis Task: –To synthesize shapes in different phenotypes and poses with the mean shape μ V.

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Shape Synthesis: Demo Shape Generator Pose Generator (Running)

Experiments on Single View Reconstruction Training dataset: –Shark data: M S : 11 3D models of different shark species. M A : 11-frame tail-waving sequence from an animatable 3D MEX model. –Human data: M S : CAESAR dataset. M A : Animations of different 3D poses of Sydney in Poser 7. Testing: –Internet images (22 sharks and 20 humans in different poses and camera viewpoints) Segmentation: GrabCut [Rother’04]

Experiments on Single View Reconstruction Sharks:

Experiments on Single View Reconstruction Humans:

Experiments on Single View Reconstruction Examples of multi-modality

Experiments on Single View Reconstruction Qualitative Results: Precision-Recall Ratios –S F : foreground regions –S R : image projection of our result A very good approximation to the results given by parametrical models

Roadmap Brief Introductions Our Framework Experimental Results Summary

Pros and Cons: Advantages Fully data driven; Requiring no strong class- specific prior knowledge, e.g., skeleton, joint angles; Capable of modelling general categories; Compact shape representation and much lower dimensions for efficient optimization; Uncertainty measurements provided. Disadvantages Inaccurate at fine parts, e.g., hands. Lower descriptive power on poses compared with parametric model, when training instances are not enough; Training data are sometimes difficult to obtain.

Future Work A compatible framework which allows incorporating category knowledge Incorporating more cues: internal edges, texture, and colour; Multiple view settings and video sequences; 3D object recognition and action recognition tasks.

Thanks!