1. 3D Computer Vision and Video Computing 3D Vision Lecture 14 Stereo Vision (I) CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/ Capstone2004/Capstone_Sequence2004.html

2. 3D Computer Vision and Video Computing Stereo Vision n Problem l Infer 3D structure of a scene from two or more images taken from different viewpoints n Two primary Sub-problems l Correspondence problem (stereo match) -> disparity map n Similarity instead of identity n Occlusion problem: some parts of the scene are visible only in one eye l Reconstruction problem -> 3D n What we need to know about the cameras’ parameters n Often a stereo calibration problem n Lectures on Stereo Vision l Stereo Geometry – Epipolar Geometry (*) l Correspondence Problem (*) – Two classes of approaches l 3D Reconstruction Problems – Three approaches

3. 3D Computer Vision and Video Computing A Stereo Pair n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D CMU CIL Stereo Dataset : Castle sequence http://www-2.cs.cmu.edu/afs/cs/project/cil/ftp/html/cil-ster.html ? 3D?

4. 3D Computer Vision and Video Computing More Images… n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D

5. 3D Computer Vision and Video Computing More Images… n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D

6. 3D Computer Vision and Video Computing More Images… n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D

7. 3D Computer Vision and Video Computing More Images… n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D

8. 3D Computer Vision and Video Computing More Images… n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D

9. 3D Computer Vision and Video Computing Lecture Outline n A Simple Stereo Vision System l Disparity Equation l Depth Resolution l Fixated Stereo System n Zero-disparity Horopter n Epipolar Geometry l Epipolar lines – Where to search correspondences n Epipolar Plane, Epipolar Lines and Epipoles n http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html l Essential Matrix and Fundamental Matrix n Computing E & F by the Eight-Point Algorithm n Computing the Epipoles n Stereo Rectification

10. 3D Computer Vision and Video Computing Stereo Geometry n Converging Axes – Usual setup of human eyes n Depth obtained by triangulation n Correspondence problem: p l and p r correspond to the left and right projections of P, respectively. P(X,Y,Z)

11. 3D Computer Vision and Video Computing A Simple Stereo System Z w =0 LEFT CAMERA Left image: reference Right image: target RIGHT CAMERA Elevation Z w disparity Depth Z baseline

12. 3D Computer Vision and Video Computing Disparity Equation P(X,Y,Z) p l (x l,y l ) Optical Center O l f = focal length Image plane LEFT CAMERA B = Baseline Depth Stereo system with parallel optical axes f = focal length Optical Center O r p r (x r,y r ) Image plane RIGHT CAMERA Disparity: dx = x r - x l

13. 3D Computer Vision and Video Computing Disparity vs. Baseline P(X,Y,Z) p l (x l,y l ) Optical Center O l f = focal length Image plane LEFT CAMERA B = Baseline Depth f = focal length Optical Center O r p r (x r,y r ) Image plane RIGHT CAMERA Disparity dx = x r - x l Stereo system with parallel optical axes

14. 3D Computer Vision and Video Computing Depth Accuracy n Given the same image localization error l Angle of cones in the figure n Depth Accuracy (Depth Resolution) vs. Baseline l Depth Error  1/B (Baseline length) l PROS of Longer baseline, n better depth estimation l CONS n smaller common FOV n Correspondence harder due to occlusion n Depth Accuracy (Depth Resolution) vs. Depth l Disparity (>0)  1/ Depth l Depth Error  Depth 2 l Nearer the point, better the depth estimation n An Example l f = 16 x 512/8 pixels, B = 0.5 m l Depth error vs. depth Z2Z2 Two viewpoints Z2>Z1Z2>Z1 Z1Z1 Z1Z1 OlOl OrOr Absolute error Relative error

15. 3D Computer Vision and Video Computing Stereo with Converging Cameras n Stereo with Parallel Axes l Short baseline n large common FOV n large depth error l Long baseline n small depth error n small common FOV n More occlusion problems n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases FOV Leftright

16. 3D Computer Vision and Video Computing Stereo with Converging Cameras n Stereo with Parallel Axes l Short baseline n large common FOV n large depth error l Long baseline n small depth error n small common FOV n More occlusion problems n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases FOV Leftright

17. 3D Computer Vision and Video Computing Stereo with Converging Cameras n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases n Disparity properties l Disparity uses angle instead of distance l Zero disparity at fixation point n and the Zero-disparity horopter l Disparity increases with the distance of objects from the fixation points n >0 : outside of the horopter n <0 : inside the horopter n Depth Accuracy vs. Depth l Depth Error  Depth 2 l Nearer the point, better the depth estimation FOV Leftright  Fixation point

18. 3D Computer Vision and Video Computing  Stereo with Converging Cameras n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases n Disparity properties l Disparity uses angle instead of distance l Zero disparity at fixation point n and the Zero-disparity horopter l Disparity increases with the distance of objects from the fixation points n >0 : outside of the horopter n <0 : inside the horopter n Depth Accuracy vs. Depth l Depth Error  Depth 2 l Nearer the point, better the depth estimation Leftright Fixation point ll rr  r =  l d  = 0 Horopter

19. 3D Computer Vision and Video Computing  Stereo with Converging Cameras n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases n Disparity properties l Disparity uses angle instead of distance l Zero disparity at fixation point n and the Zero-disparity horopter l Disparity increases with the distance of objects from the fixation points n >0 : outside of the horopter n <0 : inside the horopter n Depth Accuracy vs. Depth l Depth Error  Depth 2 l Nearer the point, better the depth estimation Leftright Fixation point ll rr  r >  l d  > 0 Horopter

20. 3D Computer Vision and Video Computing Stereo with Converging Cameras n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases n Disparity properties l Disparity uses angle instead of distance l Zero disparity at fixation point n and the Zero-disparity horopter l Disparity increases with the distance of objects from the fixation points n >0 : outside of the horopter n <0 : inside the horopter n Depth Accuracy vs. Depth l Depth Error  Depth 2 l Nearer the point, better the depth estimation Leftright Fixation point LL rr  r <  l d  < 0 Horopter

21. 3D Computer Vision and Video Computing Stereo with Converging Cameras n Two optical axes intersect at the Fixation Point converging angle  l The common FOV Increases n Disparity properties l Disparity uses angle instead of distance l Zero disparity at fixation point n and the Zero-disparity horopter l Disparity increases with the distance of objects from the fixation points n >0 : outside of the horopter n <0 : inside the horopter n Depth Accuracy vs. Depth l Depth Error  Depth 2 l Nearer the point, better the depth estimation Leftright Fixation point ll rr  d  Horopter

22. 3D Computer Vision and Video Computing Parameters of a Stereo System n Intrinsic Parameters l Characterize the transformation from camera to pixel coordinate systems of each camera l Focal length, image center, aspect ratio n Extrinsic parameters l Describe the relative position and orientation of the two cameras l Rotation matrix R and translation vector T p l p r P OlOl OrOr XlXl XrXr PlPl PrPr flfl frfr ZlZl YlYl ZrZr YrYr R, T

23. 3D Computer Vision and Video Computing Epipolar Geometry n Notations l P l =(X l, Y l, Z l ), P r =(X r, Y r, Z r ) n Vectors of the same 3-D point P, in the left and right camera coordinate systems respectively l Extrinsic Parameters n Translation Vector T = (O r -O l ) n Rotation Matrix R l p l =(x l, y l, z l ), p r =(x r, y r, z r ) n Projections of P on the left and right image plane respectively n For all image points, we have z l =f l, z r =f r p l p r P OlOl OrOr XlXl XrXr PlPl PrPr flfl frfr ZlZl YlYl ZrZr YrYr R, T

24. 3D Computer Vision and Video Computing Epipolar Geometry n Motivation: where to search correspondences? l Epipolar Plane n A plane going through point P and the centers of projection (COPs) of the two cameras l Conjugated Epipolar Lines n Lines where epipolar plane intersects the image planes l Epipoles n The image in one camera of the COP of the other n Epipolar Constraint l Corresponding points must lie on conjugated epipolar lines p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

25. 3D Computer Vision and Video Computing Essential Matrix n Equation of the epipolar plane l Co-planarity condition of vectors P l, T and P l -T n Essential Matrix E = RS l 3x3 matrix constructed from R and T (extrinsic only) n Rank (E) = 2, two equal nonzero singular values Rank (R) =3 Rank (S) =2

26. 3D Computer Vision and Video Computing Essential Matrix n Essential Matrix E = RS l A natural link between the stereo point pair and the extrinsic parameters of the stereo system n One correspondence -> a linear equation of 9 entries n Given 8 pairs of (pl, pr) -> E l Mapping between points and epipolar lines we are looking for n Given p l, E -> p r on the projective line in the right plane n Equation represents the epipolar line of either pr (or pl) in the right (or left) image n Note: l pl, pr are in the camera coordinate system, not pixel coordinates that we can measure

27. 3D Computer Vision and Video Computing Fundamental Matrix n Mapping between points and epipolar lines in the pixel coordinate systems l With no prior knowledge on the stereo system n From Camera to Pixels: Matrices of intrinsic parameters n Questions: l What are fx, fy, ox, oy ? l How to measure p l in images? Rank (M int ) =3

28. 3D Computer Vision and Video Computing Fundamental Matrix n Fundamental Matrix l Rank (F) = 2 l Encodes info on both intrinsic and extrinsic parameters l Enables full reconstruction of the epipolar geometry l In pixel coordinate systems without any knowledge of the intrinsic and extrinsic parameters l Linear equation of the 9 entries of F

29. 3D Computer Vision and Video Computing Computing F: The Eight-point Algorithm n Input: n point correspondences ( n >= 8) l Construct homogeneous system Ax= 0 from n x = (f 11,f 12,,f 13, f 21,f 22,f 23 f 31,f 32, f 33 ) : entries in F n Each correspondence give one equation n A is a nx9 matrix l Obtain estimate F^ by SVD of A n x (up to a scale) is column of V corresponding to the least singular value l Enforce singularity constraint: since Rank (F) = 2 n Compute SVD of F^ n Set the smallest singular value to 0: D -> D’ n Correct estimate of F : n Output: the estimate of the fundamental matrix, F’ n Similarly we can compute E given intrinsic parameters

30. 3D Computer Vision and Video Computing Locating the Epipoles from F n Input: Fundamental Matrix F l Find the SVD of F l The epipole e l is the column of V corresponding to the null singular value (as shown above) l The epipole e r is the column of U corresponding to the null singular value n Output: Epipole e l and e r e l lies on all the epipolar lines of the left image F is not identically zero For every p r p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

31. 3D Computer Vision and Video Computing Stereo Rectification n Rectification l Given a stereo pair, the intrinsic and extrinsic parameters, find the image transformation to achieve a stereo system of horizontal epipolar lines l A simple algorithm: Assuming calibrated stereo cameras p’ l r P OlOl OrOr X’ r PlPl PrPr Z’ l Y’ l Y’ r TX’ l Z’ r n Stereo System with Parallel Optical Axes n Epipoles are at infinity n Horizontal epipolar lines

32. 3D Computer Vision and Video Computing Stereo Rectification n Algorithm l Rotate both left and right camera so that they share the same X axis : O r -O l = T l Define a rotation matrix R rect for the left camera l Rotation Matrix for the right camera is R rect R T l Rotation can be implemented by image transformation p l p r P OlOl OrOr XlXl XrXr PlPl PrPr ZlZl YlYl ZrZr YrYr R, T TX’ l X l ’ = T, Y l ’ = X l ’xZ l, Z’ l = X l ’xY l ’

33. 3D Computer Vision and Video Computing Stereo Rectification n Algorithm l Rotate both left and right camera so that they share the same X axis : O r -O l = T l Define a rotation matrix R rect for the left camera l Rotation Matrix for the right camera is R rect R T l Rotation can be implemented by image transformation p l p r P OlOl OrOr XlXl XrXr PlPl PrPr ZlZl YlYl ZrZr YrYr R, T TX’ l X l ’ = T, Y l ’ = X l ’xZ l, Z’ l = X l ’xY l ’

34. 3D Computer Vision and Video Computing Stereo Rectification n Algorithm l Rotate both left and right camera so that they share the same X axis : O r -O l = T l Define a rotation matrix R rect for the left camera l Rotation Matrix for the right camera is R rect R T l Rotation can be implemented by image transformation ZrZr p’ l r P OlOl OrOr X’ r PlPl PrPr Z’ l Y’ l Y’ r R, T TX’ l T’ = (B, 0, 0), P’ r = P’ l – T’

35. 3D Computer Vision and Video Computing Next n Two Primary Sub-problems in Stereo Vision l Correspondence problem l 3D reconstruction from stereo Stereo Vision (II)