1. Accurate, Dense and Robust Multi-View Stereopsis Yasutaka Furukawa and Jean Ponce Presented by Rahul Garg and Ryan Kaminsky

2. Agenda Problem Statement Multi-view Stereo Taxonomy Algorithm Results Comparison to other works Questions

3. Problem Statement Multi-view Stereo –Dense shape reconstruction from multiple views +++ =

4. Multi-View Stereo Taxonomy Scene Representation Photoconsistency Measure Visibility Model Shape Prior Reconstruction algorithm Initialization S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski

5. Scene Representation –Geometry on 3D grid Voxels, Level sets –Polygon Mesh Set of planar facets –Depth Map Image that stores depth per pixel

6. Photoconsistency Measure Definition: Measures visual compatibility of reconstruction with input images –Scene Space Project part of reconstruction into images, measure closeness Measures: Variance, sum of squared distances, normalized cross-correlation –Image Space Use scene geometry to transform image to different view, measure error of predicted vs. actual (prediction error)

7. Visibility Model Definition: Views to consider when evaluating photo consistency –Geometric Explicitly model geometry of the scene –Quasi-Geometric Approximate geometric reasoning –Outlier based approaches Treat occlusions as outliers

8. Shape Prior Definition: Additional constraints or assumptions about reconstruction –Minimal Surfaces Level sets, Min-cut –Maximal Surfaces Voxel coloring, space carving –Local Measures Assume local smoothness on nearby pixels

9. Reconstruction Algorithm Optimize cost function –Voxels, graph cut, level sets, meshes A set of consistent depth maps Feature extraction, matching, surface fitting

10. Initialization Definition: Constraints on scene geometry –Bounding box or volume –Visual hull –Range of disparity

11. Overview of Algorithm input image detected reconstructed final patches polygonal surface features patches after after expansion from reconstructed the initial and filtering patches matching

12. Algorithm Block Diagram Initialization ExpansionFilter Feature Detection Reconstruction Patch Model

13. Init Detect features using Harris Corner and DoG Feature matching to generate sparse set of patches

14. Patch Models R(p): Most closely associated image with p S(p): Images where p should be visible T(p): Images where p is truly visible

15. β pixels Epipolar line c(p): from triangulation n(p): Direction of optical ray from c(p) to O

16. Normalized Cross Correlation (NCC) Optimization step: Maximizing the average NCC score where is the mean of the feature and is the mean of f(x,y) in the region under the feature.

17. Patch Expansion Expand patches along tangential planes into empty areas. Optimize for normal and center and add if photometric constraints are satisfied in at least k images.

18. Filtering Analyzing visibility consistency

19. Filtering (Contd.) Local smoothness constraint : Remove patches for which proportion of neighboring patches with tangential plane “nearly” parallel is less than ε

20. Polygonal Surface Reconstruction Initialize using convex hull of patches Iteratively deform/snap to the patch model using two kinds of forces –Smoothness term –Photometric Consistency term S : Current surface S* : True surface n(v) : Normal at v Π(v) : Set of patches compatible with v d(v) : Distance between S and S*

21. Algorithm Taxonomy Categorization Scene Representation –Depth Map + Mesh Photoconsistency Measure –NCC Shape Prior –Assume local smoothness Reconstruction –Feature extraction, depth maps, optimization over patches Initialization –None

22. Results Patch ModelPolygonal Surface Model

23. Results (Contd.)

24. Evaluation on vision.middlebury.edu Temple (# of views)Dino (# of views) Full (312) Ring (47) Sparse (16) Full (312) Ring (47) Sparse (16) This paper0.540.550.620.320.330.42 Goesele et. al.0.420.61*0.87*0.460.46*0.56* Hernandez et. al.0.360.520.750.490.450.60 Accuracy Measure: Distance d in mm which brings 90% of the reconstruction within ground truth * Old Results

25. Results (Contd.) Handle occlusions/obstacles

26. Similar Approaches Setup similar to Goesele et al. (ICCV’07) – initialize patches, expand and optimize for position and normal This PaperGoesele et. al. Initialize patches using triangulated points Initialize using Structure from Motion features Explicit occlusion handlingOcclusion handling through outlier removal and view selection, prioritize patch candidates for expansion

27. Questions Pose the problem as an optimization problem simultaneously accounting for local smoothness, photo consistency, occlusion Convergence of Expand/Filter – do more iterations lead to better reconstructions? Occlusion/Outlier handling – results on more datasets Advantages of patch model – Adaptive Resolution, generalizes to large number of object classes