Epipolar geometry Class 5

Geometric Computer Vision course schedule (tentative) LectureExercise Sept 16Introduction- Sept 23Geometry & Camera modelCamera calibration Sept 30Single View Metrology (Changchang Wu) Measuring in images Oct. 7Feature Tracking/MatchingCorrespondence computation Oct. 14Epipolar GeometryF-matrix computation Oct. 21Shape-from-SilhouettesVisual-hull computation Oct. 28Multi-view stereo matchingProject proposals Nov. 4Structure from motion and visual SLAMPapers Nov. 11Multi-view geometry and self-calibration Papers Nov. 18Shape-from-XPapers Nov. 25Structured light and active range sensing Papers Dec. 23D modeling, registration and range/depth fusion (Christopher Zach?) Papers Dec. 9Appearance modeling and image- based rendering Papers Dec. 16Final project presentations

(i)Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the corresponding point x’ in the second image? (ii)Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i=1,…,n, what are the cameras P and P’ for the two views? (iii)Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre-image) X in space? Three questions: Two-view geometry

The epipolar geometry C,C’,x,x’ and X are coplanar

The epipolar geometry What if only C,C’,x are known?

The epipolar geometry All points on  project on l and l’

The epipolar geometry Family of planes  and lines l and l’ Intersection in e and e’

The epipolar geometry epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs)

Example: converging cameras

Example: motion parallel with image plane (simple for stereo  rectification)

Example: forward motion e e’

The fundamental matrix F algebraic representation of epipolar geometry we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

The fundamental matrix F geometric derivation mapping from 2-D to 1-D family (rank 2)

The fundamental matrix F algebraic derivation (note: doesn’t work for C=C’  F=0)

The fundamental matrix F correspondence condition The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x ’ in the two images

The fundamental matrix F F is the unique 3x3 rank 2 matrix that satisfies x’ T Fx=0 for all x↔x’ (i)Transpose: if F is fundamental matrix for (P,P’), then F T is fundamental matrix for (P’,P) (ii)Epipolar lines: l’=Fx & l=F T x’ (iii)Epipoles: on all epipolar lines, thus e’ T Fx=0,  x  e’ T F=0, similarly Fe=0 (iv)F has 7 d.o.f., i.e. 3x3-1(homogeneous)-1(rank2) (v)F is a correlation, projective mapping from a point x to a line l’=Fx (not a proper correlation, i.e. not invertible)

Fundamental matrix for pure translation

General motion Pure translation for pure translation F only has 2 degrees of freedom

The fundamental matrix F relation to homographies valid for all plane homographies

The fundamental matrix F relation to homographies requires

Projective transformation and invariance Derivation based purely on projective concepts F invariant to transformations of projective 3-space unique not unique canonical form

Projective ambiguity of cameras given F previous slide: at least projective ambiguity this slide: not more! Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that P=HP and P’=HP’ ~ ~ ~ lemma: (22-15=7, ok)

The projective reconstruction theorem If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent allows reconstruction from pair of uncalibrated images!

Canonical cameras given F Possible choice: Canonical representation:

C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 m1m1 m2m2 C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 m1m1 L1L1 m2m2 L2L2 M l2l2 lT1lT1 Epipolar geometry Canonical representation:

Epipolar geometry? courtesy Frank Dellaert

Other entities besides points? Lines give no constraint for two view geometry (but will for three and more views) Curves and surfaces yield some constraints related to tangency (e.g. Sinha et al. CVPR’04)

Computation of F Linear (8-point) Minimal (7-point) Robust (RANSAC) Non-linear refinement (MLE, …) Practical approach

Epipolar geometry: basic equation separate known from unknown (data) (unknowns) (linear)

~10000 ~100 1 ! Orders of magnitude difference between column of data matrix  least-squares yields poor results the NOT normalized 8-point algorithm

Transform image to ~[-1,1]x[-1,1] (0,0) (700,500) (700,0) (0,500) (1,-1) (0,0) (1,1)(-1,1) (-1,-1) normalized least squares yields good results (Hartley, PAMI´97) the normalized 8-point algorithm

the singularity constraint SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation

the minimum case – 7 point correspondences one parameter family of solutions but F 1 + F 2 not automatically rank 2

F1F1 F2F2 F 33 F 7pts (obtain 1 or 3 solutions) (cubic equation) the minimum case – impose rank 2 Compute possible as eigenvalues of (only real solutions are potential solutions)

Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis) Automatic computation of F RANSAC

restrict search range to neighborhood of epipolar line (e.g.  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

(Mostly) planar scene (see next slide) Absence of sufficient features (no texture) Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition Issues:

Computing F for quasi-planar scenes QDEGSAC 17% success for RANSAC 100% for QDEGSAC #inliers data rank 337 matches on plane, 11 off plane %inclusion of out-of-plane inliers

geometric relations between two views is fully described by recovered 3x3 matrix F two-view geometry