3D Reconstruction Using Image Sequence CSCI480/582 Lecture 32 Chap 6.1 3D Reconstruction Using Image Sequence Apr, 15, 2009

Outline The 2D-3D problem A perfect stereo vision condition The critical issues in real applications Camera calibration Finding correspondence between images

The 2D-3D Problem Given an multi-camera images of a static scene, reconstruct the 3D scene Photo tourism demo Given an image sequence from a moving camera of a static scene, reconstruct the 3D scene Given an image sequence from a moving camera of an unconstrained scene, reconstruct the static content of the 3D scene Given image sequences from multiple moving cameras of an unconstrained scene, reconstruct the 3D scene

Static Scene and Multiple Camera Views An unknown static scene Several viewpoints 4 views up to several hundreds ~20-50 on average

Sample Image Sequence [Lhuillier and Quan] How to retrieve the 3D geometry of the object given these images?

A Perfect Stereo Vision Condition Two perfect pin-hole cameras with known geometries Pixel coordinates of the 3D point projected onto the image plane of two cameras The 3D coordinate of the 3D point can be calculated by a simple Triangulation

Issues in Real Applications Unknown cameras! Unknown focal point location in 3D Unknown norm vector of the imaging plane Unknown focal length Length distortions, digitization resolution, projection noise Unknown pixel coordinate! Which pixels are co-responding to the same 3D point? And we need a lot of such pixel pairs to recover enough 3D point to describe the shape of a 3D object

Camera Calibration Associate a pixel to a ray in space Extrinsic parameters camera position orientation Intrinsic parameters Focal length We need to at least know the relative camera geometry between the two images to build a virtual 3D scene 2D pixel  3D ray

The Epipolar Geometry Given XL, XR must lie on the epipolar line determined by OL, OR, X, nL, and nR The Epipolar constraints represented by the Fundamental Matrix between two cameras

The Fundamental Matrix A 3x3 matrix F which relates corresponding points in stereo images Given two homogeneous image coordinates, x and x’ Fx is the epipolar line corresponding to point x F is a rank-2 matrix, with a dof of 7

Solve Fundamental Matrix Linearly For each point correspondence (x, x’) yields one equation x’TFx = 0 As long as we have enough correspondences to determine all the unknowns in F Let x = [u, v, 1]T, and x’ = [u’, v’, 1]T be a pair of corresponding points from two stereo images, the Fundamental matrix F=(Fij)1<=i,j<=3, then the epipolar constraints can be expressed as

How to Find the Correspondences? Which subpixel locations from the two images are representing the same 3D points?

A Pair of ‘Good’ Correspondence YES The quality of correspondence matching is determined by the stability of the reconstructed point location It is even tricky to do it manually in some scenarios

A Pair of Bad Correspondence NO How can we automate the correspondence matching process robustly?

Image Feature Detection Rank 2 features: corner Rank 1 features: edge

Finding Correspondences Given two sets of features Geometry correlation Ransac: Pick the best matching that provide the smallest reconstruction cost Reconstruction cost can be designed based on transformation assumptions Texture correlation Match by evaluating the neighborhood texture features Color statistics, distributions Process mipmap to avoid local matching but global mismatch Geometry and Texture correlation Combine the geometry and texture features into a super descriptor vector Then form correlation or mismatching cost functions based on the descriptor