1. Reconstructing Building Interiors from Images Yasutaka Furukawa Brian Curless Steven M. Seitz University of Washington, Seattle, USA 2011/01/16 蔡禹婷

2. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

3. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

4. Reconstruction and Visualization of Architectural Scenes ◦ Semi-automatic(Manual )  Google Earth & Virtual Earth  Façade : Building facade made for use as a real-time video game engine environment. Google Earth 4 Virtual Earth Aerial images ◦ Automatic  Ground-level images  Aerial images: A projected image which is "floating in air", and cannot be viewed normally.

5. Reconstruction and Visualization of Architectural Scenes Difficulty ◦ Little attention paid to indoor scenes  If you walk inside your home and take photographs, generating a compelling 3D reconstruction and visualization becomes much more difficult. Google Earth 4 Aerial images Virtual Earth ? ? ? ?

6. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

7. Goal Fully automatic system for interiors / outdoors ◦ Reconstructs a simple 3D model from images ◦ Provides real-time interactive visualization

8. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

9. Challenges Reconstruction ◦ Multi-view stereo (MVS) typically produces a dense model ◦ We want the model to be  Simple for real-time interactive visualization of a large scene (e.g., a whole house)  Accurate for high-quality image-based rendering  Simple mode is effective for compelling visualization

10. Challenges Indoor Reconstruction Texture-poor surfacesComplicated visibility Prevalence of thin structures (doors, walls, tables)

11. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

12. System pipeline 3D reconstruction and visualization system for architectural scenes.

13. System pipeline Image-based SFM MVS MWS Merging

14. MVSImage-basedSFMMWSMerging System pipeline Image-based

15. System pipeline Structure-from-Motion Bundler by Noah Snavely Structure from Motion for unordered image collections WEB: http://phototour.cs.washington.edu/bundler/ MVSImage-basedSFMMWSMerging

16. System pipeline PMVS by Yasutaka Furukawa and Jean Ponce Patch-based Multi-View Stereo Software/ Multi-view Stereo MVSImage-basedSFMMWSMerging

17. System pipeline Manhattan-world Stereo MVSImage-basedSFMMWSMerging

18. System pipeline MVSImage-basedSFM Manhattan-world Stereo

19. System pipeline Manhattan-world Stereo Result MVSImage-basedSFMMWSMerging

20. System pipeline Axis-aligned depth map merging (Paper contribution) MVSImage-basedSFMMWSMerging

21. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

22. Axis-aligned Depth-map Merging Basic framework is similar to volumetric MRF

23. Axis-aligned Depth-map Merging Basic framework is similar to volumetric MRF

24. Key Feature 1 - Penalty terms

25. Binary penalty Binary encodes smoothness & data

26. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

27. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation) Weak regularization at interesting places Focus on a dense model

28. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation) Weak regularization at interesting places Focus on a dense model We want a simple model

29. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

30. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

31. Key Feature 1 - Penalty terms Unary encodes data Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

32. Key Feature 1 - Penalty terms Binary is smoothness Unary encodes data Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

33. Key Feature 1 - Penalty terms Regularization becomes weak  Dense 3D model Regularization is data-independent  Simpler 3D model Binary penalty

34. Axis-aligned Depth-map Merging Align-voxel grid with the dominant axes

35. Axis-aligned Depth-map Merging Align-voxel grid with the dominant axes Data term (unary)

36. Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary)

37. Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary)

38. Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary) Smoothness (binary)

39. Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary) Smoothness (binary)

40. Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary) Smoothness (binary) Graph-cuts

41. Key Feature 2 – Regularization For large scenes, data info are not complete Typical volumetric MRFs bias to general minimal surface We bias to piece-wise planar axis-aligned for architectural scenes

42. Key Feature 2 – Regularization

47. Same energy (ambiguous)

48. Key Feature 2 – Regularization Same energy (ambiguous) Data penalty: 0

49. Key Feature 2 – Regularization Same energy (ambiguous) Data penalty: 0 Smoothness penalty:Data penalty: 0 Smoothness penalty: 24Data penalty: 0

50. Key Feature 2 – Regularization shrinkage

51. Key Feature 2 – Regularization Axis-aligned neighborhood + Potts model Ambiguous Break ties with the minimum-volume solution Piece-wise planar axis-aligned model Axis-aligned neighborhood + Potts model Ambiguous Break ties with the minimum-volume solution Piece-wise planar axis-aligned model

52. Key Feature 3 – Sub-voxel accuracy

56. Summary of Depth-map Merging For a simple and axis-aligned model ◦ Explicit regularization in binary ◦ Axis-aligned neighborhood system & minimum-volume solution For an accurate model ◦ Sub-voxel refinement

57. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

58. Experimental results Model complexity control with parameter

59. Experimental results Qualitative comparisons with a state-of- the-art MVS approach on hall with the number of faces in parentheses.

60. Experimental results Effects of the sub-voxel refinement procedure.

61. Experimental results Effects of the minimum volume constraint.

62. Experimental results Effects of the grid pruning on running time.

63. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

64. Conclusion & Future Work Conclusion ◦ Fully automated 3D reconstruction/visualization system for architectural scenes ◦ Novel depth-map merging to produce piece- wise planar axis-aligned model with sub-voxel accuracy Future work ◦ Relax Manhattan-world assumption ◦ Larger scenes (e.g., a whole building)

65. Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference

66. Reference N. Cornelis, B. Leibe, K. Cornelis, and L. V. Gool. 3d urban scene modeling integrating recognition and reconstruction. IJCV, 78(2-3):121–141, July 2008. L. Zebedin, J. Bauer, K. Karner, and H. Bischof. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In ECCV, 2008.