Geometry Reconstruction March 22, 2007

Fundamental Matrix An important problem: Determine the epipolar geometry. That is, the correspondence between a point on one camera and its epipolar line on the other camera. p OLOL OROR Epipoles Epipolar line T

Geometry Reconstruction Use eight-point algorithm, we can recover the fundamental matrix F. Knowing the fundamental matrix is lot easier.

Geometry Reconstruction Knowing the fundamental matrix, and a pair of corresponding pixels, we would like to obtain the 3D position of the corresponding scene point. There are three cases: 1.Calibrated cameras and extrinsic parameters are known. 2.Calibrated cameras with unknown extrinsic parameters 3.Uncalibrated cameras.

Calibrated Cameras (Triangulation) The geometric reconstruction is absolute (without ambiguity). PlPl PrPr

Calibrated Camera with Unknown Extrinsic Parameters The geometric reconstruction is only up to a scale. PlPl PrPr Main point: we don’t know T (the baseline of the system) and we have no way to ascertain the scale of the scene. We have only the essential matrix or fundamental matrix to work with.

Calibrated Camera with Unknown Extrinsic Parameters p r, p l are the left and right image points in camera coordinates Intrinsic parameters allow to go from pixel coordinates to camera coordinates. We get E from a few correspondences. But E is only determined up to a scale!

Calibrated Camera with Unknown Extrinsic Parameters PlPl PrPr We have no T, no information on scale. From E: E t E = S t S= Find a set of (T, R).

Uncalibrated Cameras We have two images, and that’s it! The reconstruction is only up to a global projective transformation.

Uncalibrated Cameras The ambiguity is easy to see. Only F and p r, p l are known and F is known only up to a scale. (x l, x r are 4-by-1 vectors in homogeneous coordinates). H a nonsigular 4x4 matrix

Uncalibrated Cameras Projective Transform: Given a 3D point, x=(x 1, x 2, x 3 ). In homogenous coordinates, it is x= (x 1, x 2, x 3, 1). If Hx = (y 1, y 2, y 3, y 4 ), then the image of the 3D point x under the projective transform H is (y 1 /y 4, y 2 /y 4, y 3 /y 4 ). It is a 15-dimensional (non-linear) transformation group. It is important that we know there is ambiguity in reconstruction, but it is only up to a 15-dimensional transformation group. Ambiguity is global not local.

You have a weird camera….A better camera perhaps. Impossible result

Uncalibrated Cameras Normalize P r t to [ I 3 | 0 ]. Find a P l t that satisfy the equations above. P l = [ S F | e’ ] for some skew-symmetric matrix S and e’ the left epipole will do Let S = [ e’ ] x

More about the class 1.We will cover two (and half) more topics: Shape from shading (differential geometry), optical flows (motions) and perhaps recognition (Chapters in Horns’ book ) 2.Office hours: Normally 2-4 on Friday. But come by anytime you need to discuss issues/problems with me. (Send to see if I am in office.) 3.Assignments: To be discussed. 4.Solutions: Will be available starting today. TA has a busy semester so far. 5.Problem 4 will be available shortly ( couldn’t make the Monday deadline).

Summary

Metric Reconstruction More sophisticated method using other constraints will reduce the projective ambiguity down to a global unknown similarity transform. Assume 1.both cameras have the same intrinsic parameters 2.Sufficiently many orthogonal lines have been identified. Covered in advanced vision class.

Reconstruction From N-Views (Projective) Reconstruction from a possibly large set of images. Problem: Set of 3D points, X j Set of cameras P i For each camera, image points x j i (the input data) Find P i, X j, such that P i X j = x j i

N views and M points: Total number of parameters: 11N+3M. Number of Equations: NM With enough points and views, we have number of equations > total number of parameters. The problem is over- constrained. (What about N=2?) Reconstruction From N-Views

Tomasi and kanade

Factorization

m is the number views and n is the number of points.

SVD and factorization

Projective Factorization

Iterative Optimization

State of the Art Input: A video sequence Output: Camera Matrices and 3D locations of the points (up to a global similarity transform).

Stereo Correspondence Problem

Solving Stereo Correspondence Problem 1.Intensity Correlation 2.Edge Matching

Intensity Correlation

Edge Correlation

Why Stereo Correspondence Problem Hard? Distorted Subwindows if disparity is not constant (complicates correlation)

Why Stereo Correspondence Problem Hard?