1. © 2005 Yusuf Akgul Gebze Institute of Technology Department of Computer Engineering Computer Vision Stereo

2. © 2005 Yusuf Akgul Stereo The ability to infer information on the 3D structure and distance of a scene from two or more images taken from different viewpoints.

3. © 2005 Yusuf Akgul Two Problems Of Stereo We need to solve two problems for estimating 3D structure and distance Correspondence problem is determining which parts of the left and right (or other) images correspond to each other. Reconstruction problem uses the correspondences and the camera geometry to recover the 3D structure and the distance.

4. © 2005 Yusuf Akgul Finding Correspondences O2O2O2O2 P’ 2 Q’ 2 PQ O1O1O1O1 P’ 1 Q’ 1

5. © 2005 Yusuf Akgul Finding Correspondences:

6. © 2005 Yusuf Akgul 3D Reconstruction We must solve the correspondence problem first! O2O2O2O2 P’ 2 P O1O1O1O1 P’ 1

7. © 2005 Yusuf Akgul Correspondence and 3D reconstruction

8. © 2005 Yusuf Akgul A simple stereo system OlOlOlOl OrOrOrOr P plplplpl prprprpr T Z xlxlxlxl xrxrxrxr f Disparity: T is the stereo baseline d measures the difference in retinal position between corresponding points

9. © 2005 Yusuf Akgul A simple stereo system OlOlOlOl OrOrOrOr P plplplpl prprprpr T Z xlxlxlxl xrxrxrxr f Depth is inversely proportional to disparityDepth is inversely proportional to disparity

10. © 2005 Yusuf Akgul Simple Stereo Model

11. © 2005 Yusuf Akgul Parameters of a stereo system OlOlOlOl OrOrOrOrP plplplpl prprprpr T Z xlxlxlxl xrxrxrxr f Intrinsic:Intrinsic: f: focal length of camerasf: focal length of cameras c l and c r : principal pointsc l and c r : principal points Extrinsic:Extrinsic: T: stereo baselineT: stereo baseline Transformation between cameras for a more general configurationTransformation between cameras for a more general configuration clclclcl crcrcrcr

12. © 2005 Yusuf Akgul The Correspondence Problem Basic assumptions: –Most scene points are visible in both images –Corresponding image regions are similar These assumptions hold if: –The distance of the fixation point from the cameras is much larger than the stereo baseline: Z >> T

13. © 2005 Yusuf Akgul The Correspondence Problem Is a “search” problem: –Given an element in the left image, search for the corresponding element in the right image. We must choose: –Elements to match –A similarity measure to compare elements

14. © 2005 Yusuf Akgul Correspondence Problem Two classes of algorithms: –Correlation-based algorithms Produce a DENSE set of correspondences –Feature-based algorithms Produce a SPARSE set of correspondences

15. © 2005 Yusuf Akgul Correlation-based Algorithms Elements to be matched by a similarity measure: –Image WINDOWS of fixed size. What can we use for similarity?

16. © 2005 Yusuf Akgul Correlation Based Correspondence

17. © 2005 Yusuf Akgul Comparing Windows: =?f g Some possible measures: Mostpopular

18. © 2005 Yusuf Akgul “SSD” or “block matching” (Sum of Squared Differences) It is the most popular.

19. © 2005 Yusuf Akgul Cross-Correlation C fg Is also very popular and it is closely related to the SSD:

20. © 2005 Yusuf Akgul Finding the disparity map Inputs: –Left image I l –Right image I r Parameters that must be chosen: –Correlation Window size 2W+1 –Search Window size  –Similarity measure 

21. © 2005 Yusuf Akgul CORR_MATCHING Algorithm Let p l and p r be pixels on the I l and I r Let R(p l ) be the search window  x  on I r associated with p l Let d be the displacement between p l and a point in R(p l ). d  2W+1 plplplpl 2W+1

22. © 2005 Yusuf Akgul CORR_MATCHING Algorithm For each pixel p l =[i,j] in I l do: –For each displacement d=[d 1,d 2 ] in R(p l ) do: C(d) =  k=-W k=W  k=-W k=W  (I l (i+k,j+l),I r (i+k-d 1,j+l-d 2 )) –The disparity of p l is the vector d that maximizes C(d) over R(p l ) Output the disparity for each pixel p l

23. © 2005 Yusuf Akgul How do we set W, R and  ? W (width of the correlation window): – should be based on the “scale” of the scene. R(P l ) and  (search window): –Size should be estimated based on the range of scene distances and the baseline: Z = fT/d or d = fT/Z –The position of R(Pl) is centered around the same pixel on both images.

24. © 2005 Yusuf Akgul Feature-based Methods Conceptually very similar to Correlation- based methods, but: –They only search for correspondences of a sparse set of image features. –Correspondences are given by the most similar feature pairs. –Similarity measure must be adapted to the type of feature used.

25. © 2005 Yusuf Akgul Feature-based Methods: Features most commonly used: –Corners Similarity measured in terms of: –surrounding gray values (SSD, Cross-correlation) –location –Edges, Lines Similarity measured in terms of: –orientation –contrast –coordinates of edge or line’s midpoint –length of line

26. © 2005 Yusuf Akgul Example: Comparing lines l l and l r : line lengths  l and  r : line orientations (x l,y l ) and (x r,y r ): midpoints c l and c r : average contrast along lines  l    m  c : weights controlling influence The more similar the lines, the larger S is!

27. © 2005 Yusuf Akgul FEATURE_MATCHING Algorithm Inputs: – I l and I r –Set of features on the left and right Things that must be chosen: –Search Window –Similarity measure

28. © 2005 Yusuf Akgul FEATURE_MATCHING Algorithm For each feature f l in the left image: –Compute the similarity measure between f l and every feature in the search window R(f l ) –Select the feature in R(f l ) that maximizes the similarity measure. –Save the correspondence and the disparity of f l Output the list of correspondences and disparities.

29. © 2005 Yusuf Akgul Which method should we use? Correlation methods: –dense maps, good for surface reconstruction –Require textured images –Sensitive to illumination variations –Inadequate for very different viewpoints Feature methods: –Sparse maps, good for navigation –Require prior knowledge of type of scene –Must find features first

30. © 2005 Yusuf Akgul Stereo with Parallel Cameras FOV Leftright Stereo with Parallel Axes –Short baseline large common FOV large depth error –Long baseline small depth error small common FOV More occlusion problems

31. © 2005 Yusuf Akgul Stereo with Parallel Axes –Short baseline large common FOV large depth error –Long baseline small depth error small common FOV More occlusion problems Depth Accuracy vs. Depth –Depth Error is proportional to Depth 2 –Nearer the point, better the depth estimation FOV Leftright Stereo with Parallel Cameras

32. © 2005 Yusuf Akgul Stereo with Converging Cameras Two optical axes intersect at the Fixation Point –converging angle  –The common FOV Increases FOV Leftright  Fixation point

33. © 2005 Yusuf Akgul  Stereo with Converging Cameras Disparity properties –Disparity uses angle instead of distance –Zero disparity at fixation point and the Zero-disparity horopter Leftright Fixation point ll rr  r =  l d  = 0 Horopter

34. © 2005 Yusuf Akgul  Stereo with Converging Cameras Disparity properties –Disparity uses angle instead of distance –Zero disparity at fixation point and the Zero-disparity horopter –Disparity increases with the distance of objects from the fixation points >0 : outside of the horopter <0 : inside the horopter Leftright Fixation point ll rr  r >  l d  > 0 Horopter

35. © 2005 Yusuf Akgul Stereo with Converging Cameras Leftright Fixation point LL rr  r <  l d  < 0 Horopter Disparity properties –Disparity uses angle instead of distance –Zero disparity at fixation point and the Zero-disparity horopter –Disparity increases with the distance of objects from the fixation points >0 : outside of the horopter <0 : inside the horopter

36. © 2005 Yusuf Akgul Stereo with Converging Cameras Disparity properties –Disparity uses angle instead of distance –Zero disparity at fixation point and the Zero-disparity horopter –Disparity increases with the distance of objects from the fixation points >0 : outside of the horopter <0 : inside the horopter Depth Accuracy vs. Depth –Depth Error is proportional to Depth 2 –Nearer the point, better the depth estimation Leftright Fixation point ll rr  d  Horopter

37. © 2005 Yusuf Akgul Constraining the Search Space We could have used additional constraints in both feature based and correlation based algorithms. What constraints can we use?

38. © 2005 Yusuf Akgul Constraining the Search Space We could have used additional constraints in both feature based and correlation based algorithms. Uniqueness constraint Continuity constraint Geometric constraints (we will look at this next)

39. © 2005 Yusuf Akgul Epipolar Geometry Motivation: where to search correspondences? –Epipolar Plane A plane going through point P and the centers of projections (COPs) of the two cameras –Conjugated Epipolar Lines Lines where epipolar plane intersects the image planes –Epipoles The image of the COP of one camera in the other Epipolar Constraint –Corresponding points must lie on conjugated epipolar lines p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

40. © 2005 Yusuf Akgul Epipolar Geometry How do we find the epipolar lines? What do we need to find them? p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

41. © 2005 Yusuf Akgul Notations –P l =(X l, Y l, Z l ), P r =(X r, Y r, Z r ) Vectors of the same 3-D point P, in the left and right camera coordinate systems respectively –Extrinsic Parameters Translation Vector T = (O r -O l ) Rotation Matrix R –p l =(x l, y l, z l ), p r =(x r, y r, z r ) Projections of P on the left and right image plane respectively For all image points, we have z l =f l, z r =f r Epipolar Geometry p l p r P OlOl OrOr XlXl XrXr PlPl PrPr flfl frfr ZlZl YlYl ZrZr YrYr R, T

42. © 2005 Yusuf Akgul Essential Matrix Equation of the epipolar plane –Co-planarity condition of vectors P l, T and P l -T Essential Matrix E = RS –3x3 matrix constructed from R and T (extrinsic only) Rank (E) = 2, two equal nonzero singular values Rank (R) =3 Rank (S) =2

43. © 2005 Yusuf Akgul Essential Matrix E = RS –A natural link between the stereo point pair and the extrinsic parameters of the stereo system One correspondence -> a linear equation of 9 entries Given 8 pairs of (pl, pr) -> E –Mapping between points and epipolar lines we are looking for Given p l, E -> p r on the projective line in the right plane Equation represents the epipolar line of pr (or pl) in the right (or left) image Note: –pl, pr are in the camera coordinate system, not pixel coordinates that we can measure Essential Matrix

44. © 2005 Yusuf Akgul Fundamental Matrix Mapping between points and epipolar lines in the pixel coordinate systems –With no prior knowledge on the stereo system From Camera to Pixels: Matrices of intrinsic parameters Questions: –What are fx, fy, ox, oy ? –How to measure p l in images? Rank (M int ) =3

45. © 2005 Yusuf Akgul Fundamental Matrix –Rank (F) = 2 –Encodes info on both intrinsic and extrinsic parameters –Enables full reconstruction of the epipolar geometry –In pixel coordinate systems without any knowledge of the intrinsic and extrinsic parameters –Linear equation of the 9 entries of F

46. © 2005 Yusuf Akgul E = RS

47. © 2005 Yusuf Akgul Essential and Fundamental Matrix Properties

48. © 2005 Yusuf Akgul How Do We Estimate E and F? The idea is simple: Establish correspondences between two stereo images. Each correspondence gives us an equation Solve the linear set of equations to get the Matrix (F) elements. How do we solve a homogenous system of linear equations?

49. © 2005 Yusuf Akgul Singular Value Decomposition: –Any mxn matrix can be written as the product of three matrices n Singular values  i are fully determined by A n D is diagonal: d ij =0 if i  j; dii =  i (i=1,2,…,n) n  1   2  …   N  0 n Both U and V are not unique n Columns of each are mutual orthogonal vectors V1V1 U1U1

50. © 2005 Yusuf Akgul Singular Value Decomposition 1. Singularity and Condition Number –nxn A is nonsingular IFF all singular values are nonzero –Condition number : degree of singularity of A A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular 2. Rank of a square matrix A –Rank (A) = number of nonzero singular values 3. Inverse of a square Matrix –If A is nonsingular –In general, the pseudo-inverse of A 4. Eigenvalues and Eigenvectors –Eigenvalues of both A T A and AA T are si 2 (si > 0) –The columns of U are the eigenvectors of AA T (mxm) –The columns of V are the eigenvectors of A T A (nxn)

51. © 2005 Yusuf Akgul Homogeneous System –m equations for n unknowns x(m >= n-1) –Rank (A) = n-1 (by looking at the SVD of A) –A non-trivial solution (up to a arbitrary scale) by SVD: –Simply proportional to the eigenvector corresponding to the only zero eigenvalue of A T A (nxn matrix) Note: –All the other eigenvalues are positive because Rank (A)=n-1 –In practice, the eigenvector (i.e. v n ) corresponding to the minimum eigenvalue of A T A, i.e.  n 2 Singular Value Decomposition

52. © 2005 Yusuf Akgul Singular Value Decomposition Problem Statements –Numerical estimate of a matrix A whose entries are not independent –Errors introduced by noise alter the estimate to  Enforcing Constraints by SVD –Take orthogonal matrix A as an example –Find the closest matrix to Â, which satisfies the constraints exactly SVD of  Observation: D = I (all the singular values are 1) if A is orthogonal Solution: changing the singular values to those expected

53. © 2005 Yusuf Akgul Computing F: The Eight-point Algorithm Input: n point correspondences ( n >= 8) –Construct homogeneous system Ax= 0 from x = (f 11,f 12,,f 13, f 21,f 22,f 23 f 31,f 32, f 33 ) : entries in F Each correspondence give one equation A is a nx9 matrix –Obtain estimate F^ by SVD of A x (up to a scale) is column of V corresponding to the least singular value –Enforce singularity constraint: since Rank (F) = 2 Compute SVD of F^ Set the smallest singular value to 0: D -> D’ Correct estimate of F : Output: the estimate of the fundamental matrix, F’ Similarly we can compute E given intrinsic parameters

54. © 2005 Yusuf Akgul Locating the Epipoles from F Input: Fundamental Matrix F –Find the SVD of F –The epipole e l is the column of V corresponding to the null singular value (as shown above) –The epipole e r is the column of U corresponding to the null singular value Output: Epipole e l and e r e l lies on all the epipolar lines of the left image F is not identically zero For every p r p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

55. © 2005 Yusuf Akgul Stereo Rectification Rectification –Given a stereo pair, the intrinsic and extrinsic parameters, find the image transformation to achieve a stereo system of horizontal epipolar lines –A simple algorithm: Assuming calibrated stereo cameras p’ l r P OlOl OrOr X’ r PlPl PrPr Z’ l Y’ l Y’ r TX’ l Z’ r n Stereo System with Parallel Optical Axes n Epipoles are at infinity n Horizontal epipolar lines

56. © 2005 Yusuf Akgul Stereo Rectification

57. © 2005 Yusuf Akgul Rectification

58. © 2005 Yusuf Akgul Stereo Rectification Algorithm –Rotate both left and right camera so that they share the same X axis : O r -O l = T –Define a rotation matrix Rrect for the left camera –Rotation Matrix for the right camera is RrectR T –Rotation can be implemented by image transformation p l p r P OlOl OrOr XlXl XrXr PlPl PrPr ZlZl YlYl ZrZr YrYr R, T TX’ l X l ’ = T, Y l ’ = X l ’xZ l, Z’ l = X l ’xY l ’

59. © 2005 Yusuf Akgul Stereo Rectification p l p r P OlOl OrOr XlXl XrXr PlPl PrPr ZlZl YlYl ZrZr YrYr R, T TX’ l X l ’ = T, Y l ’ = X l ’xZ l, Z’ l = X l ’xY l ’ Algorithm –Rotate both left and right camera so that they share the same X axis : O r -O l = T –Define a rotation matrix Rrect for the left camera –Rotation Matrix for the right camera is RrectR T –Rotation can be implemented by image transformation

60. © 2005 Yusuf Akgul Stereo Rectification ZrZr p’ l r P OlOl OrOr X’ r PlPl PrPr Z’ l Y’ l Y’ r R, T TX’ l T’ = (B, 0, 0), P’ r = P’ l – T’ Algorithm –Rotate both left and right camera so that they share the same X axis : O r -O l = T –Define a rotation matrix Rrect for the left camera –Rotation Matrix for the right camera is RrectR T –Rotation can be implemented by image transformation

61. © 2005 Yusuf Akgul Stereo Rectification Read your book on how to obtain the rotation matrix for the rectification.

62. © 2005 Yusuf Akgul Epipolar Geometry: Summary Purpose –where to search correspondences Epipolar plane, epipolar lines, and epipoles –known intrinsic (f) and extrinsic (R, T) co-planarity equation –known intrinsic but unknown extrinsic essential matrix –unknown intrinsic and extrinsic fundamental matrix Rectification –Generate stereo pair (by software) with parallel optical axis and thus horizontal epipolar lines

63. © 2005 Yusuf Akgul 3D Reconstruction Problem What we have done –Correspondences using either correlation or feature based approaches –Epipolar Geometry from at least 8 point correspondences Three cases of 3D reconstruction depending on the amount of a priori knowledge on the stereo system –Both intrinsic and extrinsic known - > can solve the reconstruction problem unambiguously by triangulation –Only intrinsic known -> recovery structure and extrinsic up to an unknown scaling factor –Only correspondences -> reconstruction only up to an unknown, global projective transformation

64. © 2005 Yusuf Akgul Reconstruction by Triangulation Assumption and Problem –Under the assumption that both intrinsic and extrinsic parameters are known –Compute the 3-D location from their projections, pl and pr Solution –Triangulation: Two rays are known and the intersection can be computed –Problem: Two rays will not actually intersect in space due to errors in calibration and correspondences, and pixelization –Solution: find a point in space with minimum distance from both rays p p r P OlOl OrOr l

65. © 2005 Yusuf Akgul Reconstruction by Triangulation g

66. © 2005 Yusuf Akgul How to Get the Stereo Params from Camera Params If we know the camera parameters of both cameras (both cameras are calibrated), then we can easily calculate the stereo parameters. T and R are the parameters of the stereo system

67. © 2005 Yusuf Akgul Reconstruction up to a Scale Factor Assume that intrinsic parameters of both cameras are known Essential Matrix is known up to a scale factor (for example, estimated from the 8 point algorithm).

68. © 2005 Yusuf Akgul Reconstruction up to a Scale Factor Assumption and Problem Statement –Under the assumption that only intrinsic parameters and more than 8 point correspondences are given –Compute the 3-D location from their projections, pl and pr, as well as the extrinsic parameters Solution –Compute the essential matrix E from at least 8 correspondences –Estimate T (up to a scale and a sign) from E (=RS) using the orthogonal constraint of R, and then R End up with four different estimates of the pair (T, R) –Reconstruct the depth of each point, and pick up the correct sign of R and T. –Results: reconstructed 3D points (up to a common scale); –The scale can be determined if distance of two points (in space) are known

69. © 2005 Yusuf Akgul Reconstruction up to a Scale Factor We can get the components of T from this matrix easily.

70. © 2005 Yusuf Akgul Reconstruction up to a Scale Factor Let It can be proved that

71. © 2005 Yusuf Akgul Reconstruction up to a Scale Factor We have two choices of t, (t + and t - ) because of sign ambiguity and two choices of E, (E + and E - ). This gives us four pairs of translation vectors and rotation matrices.

72. © 2005 Yusuf Akgul Reconstruction up to a Scale Factor Given and 1.Construct the vectors w, and compute R 2.Reconstruct the Z and Z’ for each point 3.If the signs of Z and Z’ of the reconstructed points are a)both negative for some point, change the sign of and go to step 2. b)different for some point, change the sign of each entry of and go to step 1. c)both positive for all points, exit.

73. © 2005 Yusuf Akgul No Parameters Known Only correspondences -> reconstruction only up to an unknown, global projective transformation Read 7.4.3 in your book