Reconstructing Building Interiors from Images Yasutaka Furukawa Brian Curless Steven M. Seitz University of Washington, Seattle, USA 2011/01/16 蔡禹婷
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Reconstruction and Visualization of Architectural Scenes ◦ Semi-automatic(Manual ) Google Earth & Virtual Earth Façade : Building facade made for use as a real-time video game engine environment. Google Earth 4 Virtual Earth Aerial images ◦ Automatic Ground-level images Aerial images: A projected image which is "floating in air", and cannot be viewed normally.
Reconstruction and Visualization of Architectural Scenes Difficulty ◦ Little attention paid to indoor scenes If you walk inside your home and take photographs, generating a compelling 3D reconstruction and visualization becomes much more difficult. Google Earth 4 Aerial images Virtual Earth ? ? ? ?
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Goal Fully automatic system for interiors / outdoors ◦ Reconstructs a simple 3D model from images ◦ Provides real-time interactive visualization
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Challenges Reconstruction ◦ Multi-view stereo (MVS) typically produces a dense model ◦ We want the model to be Simple for real-time interactive visualization of a large scene (e.g., a whole house) Accurate for high-quality image-based rendering Simple mode is effective for compelling visualization
Challenges Indoor Reconstruction Texture-poor surfacesComplicated visibility Prevalence of thin structures (doors, walls, tables)
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
System pipeline 3D reconstruction and visualization system for architectural scenes.
System pipeline Image-based SFM MVS MWS Merging
MVSImage-basedSFMMWSMerging System pipeline Image-based
System pipeline Structure-from-Motion Bundler by Noah Snavely Structure from Motion for unordered image collections WEB: MVSImage-basedSFMMWSMerging
System pipeline PMVS by Yasutaka Furukawa and Jean Ponce Patch-based Multi-View Stereo Software/ Multi-view Stereo MVSImage-basedSFMMWSMerging
System pipeline Manhattan-world Stereo MVSImage-basedSFMMWSMerging
System pipeline MVSImage-basedSFM Manhattan-world Stereo
System pipeline Manhattan-world Stereo Result MVSImage-basedSFMMWSMerging
System pipeline Axis-aligned depth map merging (Paper contribution) MVSImage-basedSFMMWSMerging
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Axis-aligned Depth-map Merging Basic framework is similar to volumetric MRF
Axis-aligned Depth-map Merging Basic framework is similar to volumetric MRF
Key Feature 1 - Penalty terms
Binary penalty Binary encodes smoothness & data
Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)
Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation) Weak regularization at interesting places Focus on a dense model
Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation) Weak regularization at interesting places Focus on a dense model We want a simple model
Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)
Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)
Key Feature 1 - Penalty terms Unary encodes data Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)
Key Feature 1 - Penalty terms Binary is smoothness Unary encodes data Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)
Key Feature 1 - Penalty terms Regularization becomes weak Dense 3D model Regularization is data-independent Simpler 3D model Binary penalty
Axis-aligned Depth-map Merging Align-voxel grid with the dominant axes
Axis-aligned Depth-map Merging Align-voxel grid with the dominant axes Data term (unary)
Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary)
Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary)
Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary) Smoothness (binary)
Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary) Smoothness (binary)
Axis-aligned Depth-map Merging Align voxel grid with the dominant axes Data term (unary) Smoothness (binary) Graph-cuts
Key Feature 2 – Regularization For large scenes, data info are not complete Typical volumetric MRFs bias to general minimal surface We bias to piece-wise planar axis-aligned for architectural scenes
Key Feature 2 – Regularization
Same energy (ambiguous)
Key Feature 2 – Regularization Same energy (ambiguous) Data penalty: 0
Key Feature 2 – Regularization Same energy (ambiguous) Data penalty: 0 Smoothness penalty:Data penalty: 0 Smoothness penalty: 24Data penalty: 0
Key Feature 2 – Regularization shrinkage
Key Feature 2 – Regularization Axis-aligned neighborhood + Potts model Ambiguous Break ties with the minimum-volume solution Piece-wise planar axis-aligned model Axis-aligned neighborhood + Potts model Ambiguous Break ties with the minimum-volume solution Piece-wise planar axis-aligned model
Key Feature 3 – Sub-voxel accuracy
Summary of Depth-map Merging For a simple and axis-aligned model ◦ Explicit regularization in binary ◦ Axis-aligned neighborhood system & minimum-volume solution For an accurate model ◦ Sub-voxel refinement
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Experimental results Model complexity control with parameter
Experimental results Qualitative comparisons with a state-of- the-art MVS approach on hall with the number of faces in parentheses.
Experimental results Effects of the sub-voxel refinement procedure.
Experimental results Effects of the minimum volume constraint.
Experimental results Effects of the grid pruning on running time.
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Conclusion & Future Work Conclusion ◦ Fully automated 3D reconstruction/visualization system for architectural scenes ◦ Novel depth-map merging to produce piece- wise planar axis-aligned model with sub-voxel accuracy Future work ◦ Relax Manhattan-world assumption ◦ Larger scenes (e.g., a whole building)
Outline Introduction Goal Challenges System pipeline Algorithmic details (technical contribution) Experimental Results Conclusion and future work Reference
Reference N. Cornelis, B. Leibe, K. Cornelis, and L. V. Gool. 3d urban scene modeling integrating recognition and reconstruction. IJCV, 78(2-3):121–141, July L. Zebedin, J. Bauer, K. Karner, and H. Bischof. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In ECCV, 2008.