1. Using Strong Shape Priors for Multiview Reconstruction Yunda SunPushmeet Kohli Mathieu BrayPhilip HS Torr Department of Computing Oxford Brookes University

2. Objective + [Images Courtesy: M. Black, L. Sigal] Parametric Model Images Silhouettes Pose Estimate Reconstruction

3. Outline n Multi-view Reconstruction n Shape Models as Strong Priors n Object Specific MRF n Pose Estimation n Results

4. Outline n Multi-view Reconstruction n Shape Models as Strong Priors n Object Specific MRF n Pose Estimation n Results

5. Multiview Reconstruction Need for Shape Priors

6. Multiview Reconstruction n No Priors Silhouette Intersection Space Carving n Weak Priors Surface smoothness –Snow et al. CVPR 00 Photo consistency and smoothness –Kolmogorov and Zabih [ECCV 02] –Vogiatzis et al. [CVPR 05] [Image Courtesy: Vogiatzis et al.]

7. Outline n Multi-view Reconstruction n Shape Models as Strong Priors n Object Specific MRF n Pose Estimation n Results

8. Shape-Priors for Segmentation n OBJ-CUT [Kumar et al., CVPR 05] Integrate Shape Priors in a MRF n POSE-CUT [Bray et al., ECCV 06] Efficient Inference of Model Parameters

9. Parametric Object Models as Strong Priors n Layered Pictorial Structures n Articulated Models n Deformable Models

10. Outline n Multi-view Reconstruction n Shape Models as Strong Priors n Object Specific MRF n Pose Estimation and Reconstruction n Results

11. Object-Specific MRF

12. Energy Function Shape Prior Unary Likelihood Smoothness Prior x : Voxel label θ : Model Shape

13. Object-Specific MRF Shape Prior x : Voxel label θ : Model Shape : shortest distance of voxel i from the rendered model

14. Object-Specific MRF Smoothness Prior x : Voxel label θ : Model Shape Potts Model

15. Object-Specific MRF Unary Likelihood x : Voxel label θ : Model Shape : Visual Hull For a soft constraint we use a large constant K instead of infinity

16. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior Can be solved using Graph cuts [Kolmogorov and Zabih, ECCV02 ]

17. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior How to find the optimal Pose?

18. Outline n Multi-view Reconstruction n Shape Models as Strong Priors n Object Specific MRF n Pose Estimation n Results

19. Inference of Pose Parameters Rotation and Translation of Torso in X axes Rotation of left shoulder in X and Z axes

20. Inference of Pose Parameters Minimize F( ө ) using Powell Minimization Let F( ө ) = Computational Problem: Each evaluation of F( ө ) requires a graph cut to be computed. (computationally expensive!!) BUT.. Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

21. Outline n Multi-view Reconstruction n Shape Models as Strong Priors n Object Specific MRF n Pose Estimation n Results

22. Experiments n Deformable Models n Articulated Models Reconstruction Results Human Pose Estimation

23. Deformable Models n Four Cameras n 1.5 x 10 5 voxels n DOF of Model: 5 Visual Hull Our Reconstruction Shape Model

24. Articulated Models

25. n Four Cameras n 10 6 voxels n DOF of Model: 26 Shape Model Camera Setup

26. Articulated Models n 500 function evaluations of F(θ) required n Time per evaluation: 0.15 sec n Total time: 75 sec Let F( ө ) =

27. Articulated Models Visual Hull Our Reconstruction

28. Pose Estimation Results Visual Hull Reconstruction Pose Estimate

29. Pose Estimation Results n Quantitative Results 6 uniformly distributed cameras 12 degree (RMS) error over 21 joint angles

30. Pose Estimation Results n Qualitative Results

31. Pose Estimation Results Video 1, Camera 1

32. Pose Estimation Results Video 1, Camera 2

33. Pose Estimation Results Video 2, Camera 1

34. Pose Estimation Results Video 2, Camera 2

35. Future Work Use dimensionality reduction to reduce the number of pose parameters. - results in less number of pose parameters to optimize - would speed up inference High resolution reconstruction by a coarse to fine strategy Parameter Learning in Object Specific MRF

36. Thank You

37. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior +