1. Geometry Reconstruction March 22, 2007

2. Fundamental Matrix An important problem: Determine the epipolar geometry. That is, the correspondence between a point on one camera and its epipolar line on the other camera. p OLOL OROR Epipoles Epipolar line T

3. Geometry Reconstruction Use eight-point algorithm, we can recover the fundamental matrix F. Knowing the fundamental matrix is lot easier.

4. Geometry Reconstruction Knowing the fundamental matrix, and a pair of corresponding pixels, we would like to obtain the 3D position of the corresponding scene point. There are three cases: 1.Calibrated cameras and extrinsic parameters are known. 2.Calibrated cameras with unknown extrinsic parameters 3.Uncalibrated cameras.

5. Calibrated Cameras (Triangulation) The geometric reconstruction is absolute (without ambiguity). PlPl PrPr

6. Calibrated Camera with Unknown Extrinsic Parameters The geometric reconstruction is only up to a scale. PlPl PrPr Main point: we don’t know T (the baseline of the system) and we have no way to ascertain the scale of the scene. We have only the essential matrix or fundamental matrix to work with.

7. Calibrated Camera with Unknown Extrinsic Parameters p r, p l are the left and right image points in camera coordinates Intrinsic parameters allow to go from pixel coordinates to camera coordinates. We get E from a few correspondences. But E is only determined up to a scale!

8. Calibrated Camera with Unknown Extrinsic Parameters PlPl PrPr We have no T, no information on scale. From E: E t E = S t S= Find a set of (T, R).

9. Uncalibrated Cameras We have two images, and that’s it! The reconstruction is only up to a global projective transformation.

10. Uncalibrated Cameras The ambiguity is easy to see. Only F and p r, p l are known and F is known only up to a scale. (x l, x r are 4-by-1 vectors in homogeneous coordinates). H a nonsigular 4x4 matrix

11. Uncalibrated Cameras Projective Transform: Given a 3D point, x=(x 1, x 2, x 3 ). In homogenous coordinates, it is x= (x 1, x 2, x 3, 1). If Hx = (y 1, y 2, y 3, y 4 ), then the image of the 3D point x under the projective transform H is (y 1 /y 4, y 2 /y 4, y 3 /y 4 ). It is a 15-dimensional (non-linear) transformation group. It is important that we know there is ambiguity in reconstruction, but it is only up to a 15-dimensional transformation group. Ambiguity is global not local.

12. You have a weird camera….A better camera perhaps. Impossible result

14. Uncalibrated Cameras Normalize P r t to [ I 3 | 0 ]. Find a P l t that satisfy the equations above. P l = [ S F | e’ ] for some skew-symmetric matrix S and e’ the left epipole will do Let S = [ e’ ] x

18. More about the class 1.We will cover two (and half) more topics: Shape from shading (differential geometry), optical flows (motions) and perhaps recognition (Chapters 10-13 in Horns’ book ) 2.Office hours: Normally 2-4 on Friday. But come by anytime you need to discuss issues/problems with me. (Send email to see if I am in office.) 3.Assignments: To be discussed. 4.Solutions: Will be available starting today. TA has a busy semester so far. 5.Problem 4 will be available shortly ( couldn’t make the Monday deadline).

19. Summary

21. Metric Reconstruction More sophisticated method using other constraints will reduce the projective ambiguity down to a global unknown similarity transform. Assume 1.both cameras have the same intrinsic parameters 2.Sufficiently many orthogonal lines have been identified. Covered in advanced vision class.

23. Reconstruction From N-Views (Projective) Reconstruction from a possibly large set of images. Problem: Set of 3D points, X j Set of cameras P i For each camera, image points x j i (the input data) Find P i, X j, such that P i X j = x j i

24. N views and M points: Total number of parameters: 11N+3M. Number of Equations: NM With enough points and views, we have number of equations > total number of parameters. The problem is over- constrained. (What about N=2?) Reconstruction From N-Views

28. Tomasi and kanade

29. Factorization

30. m is the number views and n is the number of points.

31. SVD and factorization

32. Projective Factorization

33. Iterative Optimization

35. State of the Art Input: A video sequence Output: Camera Matrices and 3D locations of the points (up to a global similarity transform).

36. Stereo Correspondence Problem

38. Solving Stereo Correspondence Problem 1.Intensity Correlation 2.Edge Matching

39. Intensity Correlation

42. Edge Correlation

43. Why Stereo Correspondence Problem Hard? Distorted Subwindows if disparity is not constant (complicates correlation)

44. Why Stereo Correspondence Problem Hard?