Assignment 2 Compute F automatically from image pair (putative matches, 8-point, 7-point, iterative, RANSAC, guided matching) (due by Wednesday 19/03/03)

Rectification and structure computation class 15 Multiple View Geometry Comp Marc Pollefeys

Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. Three Views: Trifocal Tensor, Computing T. More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, auto- calibration, Dynamic SfM, Cheirality, Duality

Multiple View Geometry course schedule (subject to change) Jan. 7, 9Intro & motivationProjective 2D Geometry Jan. 14, 16(no class)Projective 2D Geometry Jan. 21, 23Projective 3D Geometry(no class) Jan. 28, 30Parameter Estimation Feb. 4, 6Algorithm EvaluationCamera Models Feb. 11, 13Camera CalibrationSingle View Geometry Feb. 18, 20Epipolar Geometry3D reconstruction Feb. 25, 27Fund. Matrix Comp. Mar. 4, 6Structure Comp.Planes & Homographies Mar. 18, 20Trifocal TensorThree View Reconstruction Mar. 25, 27Multiple View GeometryMultipleView Reconstruction Apr. 1, 3Bundle adjustmentPapers Apr. 8, 10Auto-CalibrationPapers Apr. 15, 17Dynamic SfMPapers Apr. 22, 24CheiralityProject Demos

Two-view geometry Epipolar geometry 3D reconstruction F-matrix comp. Structure comp.

Automatic computation of F (i)Interest points (ii)Putative correspondences (iii)RANSAC (iv) Non-linear re-estimation of F (v)Guided matching (repeat (iv) and (v) until stable)

Select strongest features (e.g. 1000/image) Feature points

Evaluate ZNCC,SSD,SAD for all features with similar coordinates Keep mutual best matches Still many wrong matches! ? Feature matching

Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis) RANSAC

restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches: guided matching

geometric relations between two views is fully described by recovered 3x3 matrix F two-view geometry

Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated) (standard approach)

Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

polar rectification: example

Example: Béguinage of Leuven Does not work with standard Homography-based approaches

Example: Béguinage of Leuven

Stereo matching attempt to match every pixel use additional constraints

Exploiting motion and scene constraints Ordering constraint Uniqueness constraint Disparity limit Disparity continuity constraint Epipolar constraint Epipolar constraint (through rectification)

Ordering constraint , , , , surface slice surface as a path occlusion right occlusion left

Uniqueness constraint In an image pair each pixel has at most one corresponding pixel In general one corresponding pixel In case of occlusion there is none

Disparity constraint surface slice surface as a path bounding box disparity band use reconsructed features to determine bounding box constant disparity surfaces

Disparity continuity constraint Assume piecewise continuous surface  piecewise continuous disparity In general disparity changes continuously discontinuities at occluding boundaries

Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

Example: reconstruct image from neighboring images

Multi-view depth fusion Compute depth for every pixel of reference image Triangulation Use multiple views Up- and down sequence Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

Point reconstruction

linear triangulation homogeneous inhomogeneous invariance? algebraic error yes, constraint no (except for affine)

geometric error possibility to compute using LM (for 2 or more points) or directly (for 2 points)

Geometric error Reconstruct matches in projective frame by minimizing the reprojection error (see Hartley&Sturm,CVIU´97) Non-iterative optimal solution

Optimal 3D point in epipolar plane Given an epipolar plane, find best 3D point for (x 1,x 2 ) x1x1 x2x2 l1l1 l2l2 l1l1 x1x1 x2x2 l2l2 x1´x1´ x2´x2´ Select closest points (x 1 ´,x 2 ´) on epipolar lines Obtain 3D point through exact triangulation Guarantees minimal reprojection error (given this epipolar plane)

Optimal epipolar plane Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative method Determine the epipolar plane for reconstruction Reconstruct optimal point from selected epipolar plane (Hartley and Sturm, CVIU´97) (polynomial of degree 6 check all minima, incl ∞ ) m1m1 m2m2 l 1  l 2  3DOF 1DOF

Reconstruction uncertainty consider angle between rays

Line reconstruction doesn‘t work for epipolar plane

Next class: Scene and plane homographies