1. 3D Surface Reconstruction from 2D Images (Survey)
(Fri)
Young Ki Baik, Computer Vision Lab.

2. 3D Surface Reconstruction from 2D Images
References
A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms
Steven M. Seitz, Richard Szeliski et. al. (CVPR 2006)
A Survey of Methods for Volumetric Scene Reconstruction
Greg Slabaugh et. al. (VG 2001) : Volume Graphics

3. 3D Surface Reconstruction from 2D Images
Contents
Introduction
Camera calibration
Shape from 2D images, techniques
Conclusion

4. 3D Surface Reconstruction from 2D Images
Introduction
Volumetric data representations
Gaining importance since their introduction in the early 70’s
→ 3D medical imaging [Greenleaf 70]

5. 3D Surface Reconstruction from 2D Images
Introduction
Volumetric data representation
The exponential growth of computational storage and processing
→ practical alternatives to surface based geometrical representation for many applications in computer graphics and scientific visualization

6. 3D Surface Reconstruction from 2D Images
Application
Reverse engineering
Augmented reality
Human computer interaction
Animation, Game
Etc.

7. 3D Surface Reconstruction from 2D Images
Methods for volumetric reconstruction
By hand
3D tool (3DMAX, MAYA, …)
Limitation
Too much time, tedious

8. 3D Surface Reconstruction from 2D Images
Methods for volumetric reconstruction
Laser scanner (Range data)
Advantage
High quality and accuracy
Limitation
Too much money
Specific configuration

9. 3D Surface Reconstruction from 2D Images
Methods for volumetric reconstruction
CCD camera (Images)
Advantage
Cheap price
Usefulness

10. 3D Surface Reconstruction from 2D Images
Condition of 3D reconstruction
3D point
3D object
mapping
Image plane
Camera
Camera
Camera system for obtaining images

11. 3D Surface Reconstruction from 2D Images
Condition of 3D reconstruction
Point correspondence
Camera parameter and motion
3D point
3D object
Camera
Camera
3D reconstruction system to make 3D object

12. 3D Surface Reconstruction from 2D Images
Camera Calibration
Camera parameters:
Extrinsic: Translation T, Rotation R.
Intrinsic: Focal Length f, image center (ox ,oy), effective pixel size (sx ,sy), radial distortion k.
Recover parameters from 3D points and their projections.
Object
View9
View1
View6
Camera Motion

13. 3D Surface Reconstruction from 2D Images
Camera Calibration
Simple overall flow
Camera is set fixed location.
Obtaining camera parameters using projected 2D image and world 3D data with Known plane or 3D rig.
We can acquire 3D volumetric representation by applying various reconstruction algorithm.

14. 3D Surface Reconstruction from 2D Images
Camera Calibration
Calibration with pattern:
Tsai’s method [Tsai87]
Zhang’s method [Zhang00]
Self-calibration [Maybank92]
Bundle adjustment [Triggs], evenly distribute errors.

15. 3D Surface Reconstruction from 2D Images
Shape from 2D Images
Shape from silhouette
Shape from structured light
Shape from Illumination
Shape from shading
Photometric stereo
Shape from Color(or intensity)
Voxel coloring
Stereo vision
Fusion method

16. 3D Surface Reconstruction from 2D Images
Shape from Silhouette(SFS)
Early works of vision
Effective method
Sculpturing a statue

17. 3D Surface Reconstruction from 2D Images
Shape from Silhouette(SFS)
- Calibrated cameras and object
- Set initial 3D volumetric region including object
- Obtain 3D volumetric data from intersecting back-projected volume
- Back-project each silhouette along the ray O1 O2 O3

18. 3D Surface Reconstruction from 2D Images
Shape from Silhouette(SFS)
Advantages:
Simple to implement and fairly robust
Fast execution
complete closed surface
→ commonly used as the effective initial boundary
Limitations:
only produced line hull
can’t detect non-convex region
sensitive to segmentation result
→ specific color is used as the background

19. 3D Surface Reconstruction from 2D Images
Shape from Structured Light
Rays coming out of light source hit the object surface and captured by image sensor (usually a camera) in a different angle. [Levoy00, Allen03]
Problem
- Optically uncooperative materials
- Scanning in the presence of occlusion
- Filling holes in dense polygon models

20. 3D Surface Reconstruction from 2D Images
Shape from Illumination
Shape from Shading
Assume distance point light source, orthographic projection, local shading and Lambertian surface
Given image intensity, determine depth by solving reflectance map in the fields of Radiometry.
Lambertian
A surface point is equally bright from all directions.
Limitation
- Do not provide qualified results

21. 3D Surface Reconstruction from 2D Images
Shape from Illumination
Photometric stereo
Advanced version of shape from shading
Method to determine surface shape using multiple images taken by varying illumination direction, while fixed camera position
Advantage
- Provide good results relative to shape from shading
Limitation
- Have to know the location of light sources

22. 3D Surface Reconstruction from 2D Images
Shape from Color
Voxel Coloring
Images can be constraints on 3D scene:
a valid 3D scene model projected must produce synthetic images same as the corresponding real input images.
SFS+color consistency
Opaque or not
Sees blue
Sees red
Sees green

23. 3D Surface Reconstruction from 2D Images
Shape from Color
Voxel Coloring (overall flow)
- Set up voxel region covering object.
- Set the camera on the fixed location.
- Place the object to the fixed location.

24. 3D Surface Reconstruction from 2D Images
Shape from Color
Voxel Coloring (overall flow)
- Iterate this algorithm about all voxel in the region.
- Judge opaqueness by thresholding variance of colors.
- Select a voxel and project onto the each image.

25. 3D Surface Reconstruction from 2D Images
Voxel Coloring
Advantages:
simple to implement and fairly robust
Limitations:
performance depends on voxel and image resolution.
→ reconstruct object in small area
→ high computational cost
occlusion and illumination problem

26. 3D Surface Reconstruction from 2D Images
Voxel coloring
More advanced algorithm
Space carving
Generalized voxel coloring
Multi-hypothesis voxel coloring

27. 3D Surface Reconstruction from 2D Images
Shape from color
Stereo vision
Features
popular method
pixel based method
mimics the behavior of human vision
apply feature matching criterion at all pixels simultaneously
search only over epipolar lines
(fewer candidate positions)
Scene object point
Left Camera
Optical axes
Epipolar lines
Epipolar plane
Image plane

28. 3D Surface Reconstruction from 2D Images
Shape from color
Stereo vision
Matching cost
Squared Intensity Differences (SD,SSD).
Absolutely Intensity Differences (AD,MSE).
Normalized Cross-correlation, normalized SSD.

29. 3D Surface Reconstruction from 2D Images
Stereo vision
Advantages
gives detailed surface estimates
covering wide area object
Building, topography
Fast execution
multi-view aggregation improves accuracy

30. 3D Surface Reconstruction from 2D Images
Stereo vision
Limitation
narrow baseline give rise to noisy estimates
fails in ~
textureless and occlusion areas
sparse, in complete surface
sensitive to non-Lambertian effects.
Other effective methods

31. 3D Surface Reconstruction from 2D Images
Fusion methods
Silhouette and Stereo Fusion for 3D Object Modeling
- Carlos Hernandez Esteban and Francis Schmitt (CVIU 2004)
# SFS +multi +stereo correlation voting +Gradient vector flow +Snake
High-Fidelity Image-Based Modeling
- Yasutaka Furukawa, Jean Ponce (CVR-TR-2006)
# SFS + wide baseline matching + propagation + Energy minimization
Multi-View Stereo Revisited
- Michael Goesele et. al. (CVPR2006)
# SFS +Stereo matching + volumetric method (range data + Level Set)
MultiView Geometry for Texture mapping 2D Images Onto 3D Range Data
- Lingyoun Liu et. Al. (CVPR2006)
# SFS +Stereo matching + volumetric method (range data)

32. 3D Surface Reconstruction from 2D Images
Trend of 3D reconstruction method
~1995 Simple algorithm
~1999 VC and variants (treating occlusion),
LevelSet, optimization
~2001 Probabilistic formulation
~2003 Non-Lambatian surface, specular surface,
textureless regions
~2006 PGM, Fusion method
(SFS + Stereo + Level set + …)

33. 3D Surface Reconstruction from 2D Images
Conclusion
Survey of methods for volumetric scene reconstruction from photographs.
States of the arts shows very good reconstruction results.
All algorithm do not solve problems yet.
occlusion ,illumination changes
non-Lambatian surface
Real data (no silhouette)
There is room for improvement