1. Computing F

2. Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.

3. Epipolar geometry: basic equation separate known from unknown (data) (unknowns) (linear) If A has rank 8  soln unique up to a scale factor For noisy data, rank of A might be larger than 8  least squares soln Minimize || Af || subject to ||f|| = 1  singular vector for smallest sing. value of A This is called “8 point” algorithm

4. the singularity constraint: SVD from linearly computed F matrix (rank 3) Replace F with F‘ that minimizes Forbenius norm || F – F‘ || F subject to det(F‘) = 0 Compute closest rank-2 approximation Two steps: (a) linear soln F obtained from sing. Vect corresp. To smallest singular value (b) replace F by F’ the closest singular matrix to F under Forbenius norm || A|| F = square root of sum of squares of a i,j

5. α F 1 + ( 1 – α ) F 2

6. the minimum case – 7 point correspondences Assume matrix A has rank 7  7 correspondences A is 7x9 ; If A rank 7  soln to Af =0 is 2D space of the form α F 1 + ( 1 – α ) F 2 where F 1 and F 2 are matrices corresponding to generators f 1 and f 2 of the right null space of A Det (F) = 0  det (α F 1 + ( 1 – α ) F 2 ) = 0  cubic polynomial in α  either one or 3 real solns  either one or 3 F This is called the “7 point “ algorithm.

7. F1F1 F2F2 F 33 F 7pts (obtain 1 or 3 solutions) (cubic equation) the minimum case – impose rank 2 Compute possible as eigenvalues of (only real solutions are potential solutions)

8. ~10000 ~100 1 ! Orders of magnitude difference Between column of data matrix  least-squares yields poor results the NOT normalized 8-point algorithm

9. Transform image to ~[-1,1]x[-1,1] (0,0) (700,500) (700,0) (0,500) (1,-1) (0,0)(1,1)(-1,1) (-1,-1) Least squares yields good results (Hartley, PAMI´97) the normalized 8-point algorithm Translate and scale each image so that the centroid of the reference points is at the origin of the coordinates and the RMS distance of the points from the origin is √2

11. algebraic minimization Find F‘ with rank 2 directly, i.e. Find singular matrix F‘ that minimizes || A F‘|| subject to ||F‘|| = 1 Arbitrary singular matrix F can be written as F = M [e] x where M is nonsingular and [e] x is any skew symmetrix matrix For now, assume e is known F = M [e] x  f = Em where f and m are vectors corresponding to F and M matrices and Minimize || A E m || subject to || Em || = 1 Apply Algorithm A.5.6 from Hartley and Zisserman

13. Geometric distance Gold standard Sampson error Symmetric epipolar distance

14. Gold standard Maximum Likelihood Estimation (= least-squares for Gaussian noise) Parameterize: Initialize: normalized 8-point, (P,P‘) from F, reconstruct X i Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book) (overparametrized) x i and x’ I measured correspondence; and true correspondence Find F to minimize

16. Gold standard Alternative, minimal parametrization (with a=1) (note (x,y,1) and (x‘,y‘,1) are epipoles) problems: a=0  pick largest of a,b,c,d to fix epipole at infinity  pick largest of x,y,w and of x’,y’,w’ 4x3x3=36 parametrizations! reparametrize at every iteration, to be sure

17. Zhang&Loop’s approach CVIU’01

18. First-order geometric error (Sampson error) (one eq./point  JJ T scalar) (problem if some x is located at epipole) advantage: no subsidiary variables required

19. Symmetric epipolar error Minimize distance of a point from its projected epipolar line

20. Some experiments: Epipoles far from center of image

21. Some experiments: Epipoles in the image Epipoles close to image

22. Some experiments: Matched points known extremely accurately

23. Algorithm comparison Normalized 8 point (alg. 11.1) Min. Algebraic error while imposing singularlity (Alg. 11.2) Gold standard algorithm (Alg. 11.3) N = total number of correspondences Choose n out of N points randomly 100 times Find average residual error vs. n Squared distance between a point’s epipolar line and the other image averaged over all N This is not directly minimized by any of the 3 algorithms

24. Some experiments: (for all points!) Residual error:

25. Recommendations: 1.Do not use unnormalized algorithms 2.Quick and easy to implement: 8-point normalized (alg. 11.1) 3.Better: enforce rank-2 constraint during minimization (alg. 11.2) 4.Best: Maximum Likelihood Estimation (alg. 11.3) (minimal parameterization, sparse implementation)

26. Automatic computation of F (i)Interest points (ii)Putative correspondences (iii)RANSAC (iv) Non-linear re-estimation of F (v)Guided matching (repeat (iv) and (v) until stable)

27. Extract feature points to relate images Required properties: Well-defined (i.e. neigboring points should all be different) Stable across views (i.e. same 3D point should be extracted as feature for neighboring viewpoints) Feature points

28. homogeneous edge corner If M has two small eigen value  homogeneous; if one small and one large eigenvalue  edge; if two large eigenvalues  corner (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points Feature = local maxima (subpixel) of F( 1, 2 ) Feature points

29. Select strongest features (e.g. 1000/image) Feature points

30. Evaluate NCC for all features with similar coordinates Keep mutual best matches Still many wrong matches! ? Feature matching

31. 0.96-0.40-0.16-0.390.19 -0.050.75-0.470.510.72 -0.18-0.390.730.15-0.75 -0.270.490.160.790.21 0.080.50-0.450.280.99 1 5 2 4 3 15 2 4 3 Gives satisfying results for small image motions Feature example

32. Requirement to cope with larger variations between images Translation, rotation, scaling Foreshortening Non-diffuse reflections Illumination geometric transformations photometric changes Wide-baseline matching…

33. Wide baseline matching for two different region types (Tuytelaars and Van Gool BMVC 2000) Wide-baseline matching…

34. restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

35. Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples51335106382 Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis) RANSAC

36. How many samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99 proportion of outliers e s5%10%20%25%30%40%50% 2235671117 33479111935 435913173472 54612172657146 64716243797293 748203354163588 8592644782721177

39. Absence of sufficient features (no texture) Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition More problems:

40. geometric relations between two views is fully described by recovered 3x3 matrix F two-view geometry

41. Special case: Enforce constraints for optimal results: Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof)  compute essential matrix; but for E, two singular values are equal.

42. Other entities? Lines give no constraint for two view geometry (but will for three and more views) Curves and surfaces yield some constraints related to tangency

43. Degenerate cases Planar scene Pure rotation No unique solution Remaining DOF filled by noise Use simpler model (e.g. homography) Model selection (Torr et al., ICCV´98, Kanatani, Akaike) Compare H and F according to expected residual error (compensate for model complexity) Degenerate cases:

44. The envelope of epipolar lines What happens to an epipolar line if there is noise? Monte Carlo

45. n =10 n =15 n =25 n =50

46. Next class: image pair rectification reconstructing points and lines