1 Final exam of Comp300a Venue: LG1 Time: 8h30—10h30, 31 May 2003, Sat. Content: everything in these slides + projective geometry before midterm Bring: pen, student ID Open-book exam: you can bring whatever you want

2 Stereo vision and two-view geometry The goal of a stereo system is to get 3D information A stereo system consists at least of two ‘converging’ cameras rigidly attached All final exam materials are contained in this set of slides Chapter 7 of the textbook

3 One real example: stereo geometry: epipolar geometry geometric relation between two images correspondence pixels (pts) in different images from the same pt reconstruction (triangulation) 3d coordinate of the pt Three topics:

4 Intuitive epipolar geomegtry Entirely characterized by the so-called epipolar geometry Geometric concepts: epipole: the image of the other camera center epipolar plane: plane defined by the two camera centers and the space pt epipolar lines: intersection of the epipolar plane and image plane pencil of epipolar lines and planes baseline: distance between two camera centers

5 u O u’ O epipole epipolar line epipolar plane

6 One real example of epipolar lines:

7 Algebraic characterisation of the epipolar geometry: the fundamental matrix Given a correspondence pair u and u’, where F is 3 by 3 Proof sketch: follow the geometric construction. compute the epipole in the second image compute the pt at infinity (or ray direction), reproject it onto the second define the epipolar line by these two pts use anti-symmetric matrix for cross-product Assume the camera projection matrices are P=(I 0) and P’=(A a), it can be shown that F = [a] A. The procedure is the same even if P is of general form.

8 rank of F ker(F) how many d.o.f. l’=F u l = … Properties of the fundamental matrix F:

9 Stereo Vision by ‘traditional’ calibrated approach calibration of each camera w.r.t. the same object: P and P’ (optional) rectification disparity map using F 3D reconstruction Traditional (calibrated) stereo approach: Correspondence using F (computed from P and P’) two (or more) cameras rigidly attached = stereo rig = stereo system

10 Obtain F from the given P and P’:

11 Correspondence (discussed later) 3D reconstruction : trianglulation Same equation as the calibration, but unknowns are now xi, yi, zi instead of cij

12 u O u’ O’ Triangulation:

13 ‘Modern’ uncalibrated approach: Epipolar geometry by point correspondences – two-view geometry Because of NB: it is more powerful, ‘calibration’ needs 3d info, point-correspondence does not, but not 3d reconstruction

14 8 pts algorithm 7 pts algorithm (minimal data) (optimal and robust sol.) Givencompute F

15 8-point algorithm (unstable) expand rewrite for N points rewrite

16 linear sol by svd with ||f||=1: f=v9 F’, rank enforcement afterwards by svd!

17 7-point algorithm one parameter solution by svd from the vanishing determinant, get a cubic equation

18 Normalisation 8 pt algorithm To make the average point as close as possible to (1,1,1)! normalisation by transformations linear solution for rank enforcement denormalisation Warning: unnormalised 8 pt algorithm is unstable!!!

19 Data normalisation: each image data is normlised independently!

20 Summary (or a unified view) of all methods of computation of the fundamental matrix

21 Stereo correspondence

22 disparity: difference in image position of the same space pt disparity map: dense pixel-to-pixel corrrespondences stereo rectification: make the epipolar lines horizontal an option to speed up the computation of disparity map The epipolar geometry gives only a constraint, but not yet a unique solution to the question: where is the corresponding point in the second image of a given point in the first image?

23 Rectification of a stereo pair of images: two images are transformed (by a projective transformation in image plane or by a camera rotation around the center).

24 New rectified image plane equivalent to a plane parallel to the base line T and T’ can be computed from F, but many possibilities only an option, simplify the computation!

25 or Very often ZNCC (Zero Normalized Cross Correlation), on normalised images instead of I and I’, Matching by correlation: Convert all (2n+1)(2n+1) elements from a matrix into a vector of dim (2n+1)(2n+1)

26 Two points u and u’ are in correspondence if ZNCC(u,u’) is big enoug (close to 1) dist(u’, Fu) is small enough (a few pixels)

27 Correspondence by correlation: For each point u, compute all correlations in a neighborhood u+d with a window size s Take the pixel having the highest correlation score as the correspondence Cross-validate the correspondence in the opposite direction from the second to the first image Correlation window neighborhood Cross-validate

28 When applied to ‘interest points’, sparse correspondence When applied to every pixel, dense disparity map

29 Using more cameras to remove match ambiguity: a system of 3 cameras 1 2 3

30 What can we do more with F? Without calibration, what can we get? When calibrated, essential matrix, its decomposition

31 From uncalibrated F Calibrated E

32 Essential matrix: fundamental matrix for the calibrated points The extra algebraic constraint: the equal singular values (more complicated) for E Decomposition of the essential matrix into R and t Relationship between E and F E = [t] RFrom F = [a] A

33 Decomposition of E Two factorisation Two translation Twisted pair

34

35 Given internal calibration K and K’ (more advanced studies allow us to remove this step by self-calibration that we will not handle) Compute F from point correspondences Compute E Decompose E to obtain R and t Obtain P and P’ Triangulation Summary of modern two-view approach:

36 O When space points are planar, a homography relating u and u’ u O u’ x It is therefore a ‘collineation’ for COPLANAR points! Never forget the coplanar case!

37 From P=(I 0), P’=(A a) and a known plane p^Tx=0, to get H Or, the homography can be computed from at least 4 corresponding Points, do it! The homography uniquely determines point correspondences Unlike the fundamental matrix! But only for coplanar points. So that

38 Panoramic image or image mosaicing the 3D scene is planar the camera is rotating around the center (similar to rectification) Example at HK airport (virtual tour), QuicktimeVR Realviz, stitcher, step-by-step This homography leads to one important application:

39 The images are related by a homography if the 3d scene is planar:

40 x A pencil of lines cut by two lines A star of lines cut by two planes Rotating the camera around the center is equivalent to a homographical Transformation of the image plane:

41 Compute point correspondences Compute the projective transformation between the two views Warp the first image onto the second Color-blend the overlapping areas Compositing algorithm or mosacing algorithm:

42 Quick-time VR Inward-looking small object Outward-looking large-scale environnement Example of the virtual tour of HK airport:

43 Automatic computation of the fundamental matrix Chicken-egg problem: we need corresponding points to compute F, we need F to establish correspondences … Simultaneous automatic computation of correspondences and F

44 Illustrative example of fitting a line to a set of 2D points the least squares solution (orthogonal regression) is optimal when no outliers but it is becoming very fragile to outliers

45 Robust line fitting Fit a line to 2D data containing ‘bad points’---outliers Solving two pbs: 1. A line fit to the data; 2. A classification of the data into ‘inliers’ and ‘outliers’ ‘inliers’: valid or good data satisfying the ‘line’ model ‘outliers’: bad data not satisfying the model

46 randomly draw 2 data points compute a line Li from these 2 points compute the distance to the line Li for each data point determine inliers/outliers by a threshold t compute the number of inliers Si select the Li having the largest Si re-estimate the final line using all inliers How to find the best line? repeating then

47 randomly draw a sample of s data initiate the model Mi compute the distance to the model for each data pt determine inliers/outliers by the threshold t compute the size of inliers Si select the Mi having the largest Si re-estimate using all inliers Fischler and Bolles 1981 repeating then RANSAC (random sample consensus)

48 The complete algorithm of automatic computation of F: detect points of interest in each image compute the correspondences using correlation based method RANSAC using 7-pt algo. (non-linear optimal estimation on the final inliers: this is unnecessary in many cases, so just an option)

49 randomly draw a sample of 7 corresponding points compute Fi compute the distance to Fi for each corresponding pt determine inliers/outliers by the threshold t compute the size of inliers Si select the Fi having the largest Si re-estimate the final F using all inliers repeating then RANSAC using 7-pt algo to compute F

50 Outlier proportion sSample size Probability of success (only inliers) p Ex: 99.9% success rate, 50% outliers s=2, N=17 s=4, N=72 s=6, N=293 s=7, N=588 s=8, N=1177 How many times to repeat?

51 Robust statistics From least-squares method to robust statistics (Ransac,least median of squares (LMS)) Handle ‘big errors’---outliers! This is useful not only for computing F, but also for automatically computing a mosaicing of images!