1. Robot Vision SS 2007 Matthias Rüther 1 ROBOT VISION Lesson 6a: Shape from Stereo, short summary Matthias Rüther Slides partial courtesy of Marc Pollefeys Department of Computer Science University of North Carolina, Chapel Hill

2. Robot Vision SS 2007 Matthias Rüther 2 Shape from Stereo What do we want to do? Measure 3D point location from two images of the point (from different viewpoints).

3. Robot Vision SS 2007 Matthias Rüther 3 Shape from Stereo How? Acquire two images (simultaneously, if the scene is not stationary) Determine camera geometry Find homologous (corresponding) points, i.e. points in both images which are an image of the same world point. Triangulate homologous points, i.e. determine world point from homologous points and camera geometry

4. Robot Vision SS 2007 Matthias Rüther 4 (i)We have the camera geometry and an image point, how does this constrain the location of the corresponding image point in the second image? (Correspondence geometry) (ii)We have corresponding image points, how does this constrain camera geometry? (Motion) (iii)We have the camera geometry and corresponding image points, what are the 3D point coordinates? (Correspondence geometry) Questions

5. Robot Vision SS 2007 Matthias Rüther 5 What if only C, C’, x are known? Question i: Epipolar Geometry and F

6. Robot Vision SS 2007 Matthias Rüther 6 Family of planes  and lines l and l’ Intersection in e and e’ Epipolar Geometry

7. Robot Vision SS 2007 Matthias Rüther 7 algebraic representation of epipolar geometry we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F Question i: Epipolar Geometry and F

8. Robot Vision SS 2007 Matthias Rüther 8 geometric derivation mapping from 2-D to 1-D family (rank 2) The Fundamental Matrix (F)

9. Robot Vision SS 2007 Matthias Rüther 9 algebraic derivation (note: doesn’t work for C=C’  F=0) The Fundamental Matrix (F)

10. Robot Vision SS 2007 Matthias Rüther 10 correspondence condition The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x ’ in the two images The Fundamental Matrix (F)

11. Robot Vision SS 2007 Matthias Rüther 11 Mapping of corresponding epipolar lines: derivation l,l’ epipolar lines, k line not through e  l’=F[k] x l and symmetrically l=F T [k’] x l’ (pick k=e, since e T e≠0) Epipolar Line Geometry

12. Robot Vision SS 2007 Matthias Rüther 12 Derivation based purely on projective concepts F invariant to transformations of projective 3-space unique not unique canonical form Invariance under projective transformation without derivation

13. Robot Vision SS 2007 Matthias Rüther 13 F matrix corresponds to P,P’ iff P’ T FP is skew-symmetric Possible choice: Canonical representation: Canonical cameras given F Arbitrary 3-vectorArbitrary scalar

14. Robot Vision SS 2007 Matthias Rüther 14 (only one solution where points is in front of both cameras) Reconstruction from E: four possible solutions

15. Robot Vision SS 2007 Matthias Rüther 15 C1C1 C2C2 l2l2 P l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 m1m1 m2m2 C1C1 C2C2 l2l2 P l1l1 e1e1 e2e2 m1m1 L1L1 m2m2 L2L2 M l2l2 lT1lT1 Canonical representation: Epipolar Geometry

16. Robot Vision SS 2007 Matthias Rüther 16 given x i ↔x‘ i, compute P,P‘ and X i reconstruction problem: for all i without additional information possible up to projective ambiguity 3D reconstruction of cameras and structure

17. Robot Vision SS 2007 Matthias Rüther 17 (i)Compute F from correspondences (ii)Compute camera matrices from F (iii)Compute 3D point for each pair of corresponding points computation of F use x‘ i Fx i =0 equations, linear in coeff. F 8 points (linear), 7 points (non-linear), 8+ (least-squares) computation of camera matrices use triangulation compute intersection of two backprojected rays Outline of reconstruction

18. Robot Vision SS 2007 Matthias Rüther 18 Reconstruction ambiguity: projective

19. Robot Vision SS 2007 Matthias Rüther 19 If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent  along same ray of P 2, idem for P ‘ 2 two possibilities: X 2 i = HX 1 i, or points along baseline key result: allows reconstruction from pair of uncalibrated images The projectove reconstruction theorem

20. Robot Vision SS 2007 Matthias Rüther 20

21. Robot Vision SS 2007 Matthias Rüther 21 (i)Projective reconstruction (ii)Affine reconstruction (iii)Metric reconstruction Stratified reconstruction

22. Robot Vision SS 2007 Matthias Rüther 22 use control points X E i with known coordinates to go from projective to metric (2 lin. eq. in H -1 per view, 3 for two views) Direct metric reconstruction using ground truth

23. Robot Vision SS 2007 Matthias Rüther 23 separate known from unknown (data) (unknowns) (linear) Computation of F: basic equation

24. Robot Vision SS 2007 Matthias Rüther 24 SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation Imposing the singularity constraint

25. Robot Vision SS 2007 Matthias Rüther 25

26. Robot Vision SS 2007 Matthias Rüther 26 ~10000 ~100 1 ! Orders of magnitude difference Between column of data matrix  least-squares yields poor results The NOT normalized 8-point algorithm

27. Robot Vision SS 2007 Matthias Rüther 27 Transform image to ~[-1,1]x[-1,1] (0,0) (700,500) (700,0) (0,500) (1,-1) (0,0) (1,1)(-1,1) (-1,-1) Least squares yields good results (Hartley, PAMI´97) The normalized 8-point algorithm

28. Robot Vision SS 2007 Matthias Rüther 28 Maximum Likelihood Estimation (= least-squares for Gaussian noise) Parameterize: Initialize: normalized 8-point, (P,P‘) from F, reconstruct X i Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book) (overparametrized) The Gold Standard Algorithm

29. Robot Vision SS 2007 Matthias Rüther 29 Examples

30. Robot Vision SS 2007 Matthias Rüther 30 Examples

31. Robot Vision SS 2007 Matthias Rüther 31 Examples

32. Robot Vision SS 2007 Matthias Rüther 32 1.Do not use unnormalized algorithms 2.Quick and easy to implement: 8-point normalized 3.Better: enforce rank-2 constraint during minimization 4.Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation) Recommendations

33. Robot Vision SS 2007 Matthias Rüther 33  Extract feature points to relate images  Required properties: –Well-defined (i.e. neigboring points should all be different) –Stable across views (i.e. same 3D point should be extracted as feature for neighboring viewpoints) The correspondence problem: feature points

34. Robot Vision SS 2007 Matthias Rüther 34 homogeneous edge corner M should have large eigenvalues (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points Feature = local maxima (subpixel) of F( 1, 2 ) Feature points

35. Robot Vision SS 2007 Matthias Rüther 35 Select strongest features (e.g. 1000/image) Feature points

36. Robot Vision SS 2007 Matthias Rüther 36 Evaluate NCC for all features with similar coordinates Keep mutual best matches Still many wrong matches! ? Feature matching

37. Robot Vision SS 2007 Matthias Rüther 37 0.96-0.40-0.16-0.390.19 -0.050.75-0.470.510.72 -0.18-0.390.730.15-0.75 -0.270.490.160.790.21 0.080.50-0.450.280.99 1 5 2 4 3 15 2 4 3 Gives satisfying results for small image motions Similarity Example

38. Robot Vision SS 2007 Matthias Rüther 38 Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 8 matches) Step 3.2 compute F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples51335106382 Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis) RANSAC

39. Robot Vision SS 2007 Matthias Rüther 39 restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

40. Robot Vision SS 2007 Matthias Rüther 40  Degenerate cases –Planar scene –Pure rotation  No unique solution –Remaining DOF filled by noise –Use simpler model (e.g. homography)  Model selection (Torr et al., ICCV´98, Kanatani, Akaike) –Compare H and F according to expected residual error (compensate for model complexity) Degenerate Cases

41. Robot Vision SS 2007 Matthias Rüther 41  Absence of sufficient features (no texture)  Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition More problems