1. 3D Reconstruction Using Image Sequence
CSCI480/582 Lecture 32 Chap 6.1
3D Reconstruction Using Image Sequence
Apr, 15, 2009

2. Outline The 2D-3D problem A perfect stereo vision condition
The critical issues in real applications
Camera calibration
Finding correspondence between images

3. The 2D-3D Problem
Given an multi-camera images of a static scene, reconstruct the 3D scene
Photo tourism demo
Given an image sequence from a moving camera of a static scene, reconstruct the 3D scene
Given an image sequence from a moving camera of an unconstrained scene, reconstruct the static content of the 3D scene
Given image sequences from multiple moving cameras of an unconstrained scene, reconstruct the 3D scene

4. Static Scene and Multiple Camera Views
An unknown static scene
Several viewpoints
4 views up to several hundreds
~20-50 on average

5. Sample Image Sequence [Lhuillier and Quan]
How to retrieve the 3D geometry of the object given these images?

6. A Perfect Stereo Vision Condition
Two perfect pin-hole cameras with known geometries
Pixel coordinates of the 3D point projected onto the image plane of two cameras
The 3D coordinate of the 3D point can be calculated by a simple Triangulation

7. Issues in Real Applications
Unknown cameras!
Unknown focal point location in 3D
Unknown norm vector of the imaging plane
Unknown focal length
Length distortions, digitization resolution, projection noise
Unknown pixel coordinate!
Which pixels are co-responding to the same 3D point?
And we need a lot of such pixel pairs to recover enough 3D point to describe the shape of a 3D object

8. Camera Calibration Associate a pixel to a ray in space
Extrinsic parameters
camera position
orientation
Intrinsic parameters
Focal length
We need to at least know the relative camera geometry between the two images to build a virtual 3D scene
2D pixel  3D ray

9. The Epipolar Geometry
Given XL, XR must lie on the epipolar line determined by OL, OR, X, nL, and nR
The Epipolar constraints represented by the Fundamental Matrix between two cameras

10. The Fundamental Matrix
A 3x3 matrix F which relates corresponding points in stereo images
Given two homogeneous image coordinates, x and x’
Fx is the epipolar line corresponding to point x
F is a rank-2 matrix, with a dof of 7

11. Solve Fundamental Matrix Linearly
For each point correspondence (x, x’) yields one equation x’TFx = 0
As long as we have enough correspondences to determine all the unknowns in F
Let x = [u, v, 1]T, and x’ = [u’, v’, 1]T be a pair of corresponding points from two stereo images, the Fundamental matrix F=(Fij)1<=i,j<=3, then the epipolar constraints can be expressed as

12. How to Find the Correspondences?
Which subpixel locations from the two images are representing the same 3D points?

13. A Pair of ‘Good’ Correspondence
YES
The quality of correspondence matching is determined by the stability of the reconstructed point location
It is even tricky to do it manually in some scenarios

14. A Pair of Bad CorrespondenceNO
How can we automate the correspondence matching process robustly?

15. Image Feature Detection
Rank 2 features: corner
Rank 1 features: edge

16. Finding Correspondences Given two sets of features
Geometry correlation
Ransac: Pick the best matching that provide the smallest reconstruction cost
Reconstruction cost can be designed based on transformation assumptions
Texture correlation
Match by evaluating the neighborhood texture features
Color statistics, distributions
Process mipmap to avoid local matching but global mismatch
Geometry and Texture correlation
Combine the geometry and texture features into a super descriptor vector
Then form correlation or mismatching cost functions based on the descriptor