1. Multiple View Geometry Comp 290-089 Marc Pollefeys
Computing F class 13
Multiple View Geometry
Comp
Marc Pollefeys

2. Content
Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.
Single View: Camera model, Calibration, Single View Geometry.
Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.
Three Views: Trifocal Tensor, Computing T.
More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality

3. Multiple View Geometry course schedule (subject to change)
Jan. 7, 9
Intro & motivation
Projective 2D Geometry
Jan. 14, 16
(no class)
Jan. 21, 23
Projective 3D Geometry
Jan. 28, 30
Parameter Estimation
Feb. 4, 6
Algorithm Evaluation
Camera Models
Feb. 11, 13
Camera Calibration
Single View Geometry
Feb. 18, 20
Epipolar Geometry
3D reconstruction
Feb. 25, 27
Fund. Matrix Comp.
Structure Comp.
Mar. 4, 6
Planes & Homographies
Trifocal Tensor
Mar. 18, 20
Three View Reconstruction
Multiple View Geometry
Mar. 25, 27
MultipleView Reconstruction
Bundle adjustment
Apr. 1, 3
Auto-Calibration
Papers
Apr. 8, 10
Dynamic SfM
Apr. 15, 17
Cheirality
Apr. 22, 24
Duality
Project Demos

4. Two-view geometry Epipolar geometry 3D reconstruction F-matrix comp.
Structure comp.

5. Fundamental matrix (3x3 rank 2 matrix)
Epipolar geometry l2 C1 m1 L1 m2 L2 M C2 C1 C2 l2 p l1 e1 e2 m1 L1 m2 L2 M C1 C2 l2 p l1 e1 e2
Underlying structure in set of matches for rigid scenes m1 m2
lT1 l2
Fundamental matrix (3x3 rank 2 matrix)
Computable from corresponding points
Simplifies matching
Allows to detect wrong matches
Related to calibration
Canonical representation:

6. The projective reconstruction theorem
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!

7. Objective
Given two uncalibrated images compute (PM,P‘M,{XMi})
(i.e. within similarity of original scene and cameras)
Algorithm
Compute projective reconstruction (P,P‘,{Xi})
Compute F from xi↔x‘i
Compute P,P‘ from F
Triangulate Xi from xi↔x‘i
Rectify reconstruction from projective to metric
Direct method: compute H from control points
Stratified method:
Affine reconstruction: compute p∞
Metric reconstruction: compute IAC w

8. F F,H∞ p∞ w,w’ W∞ Image information provided
View relations and projective objects
3-space objects
reconstruction ambiguity
point correspondences F
projective
point correspondences including vanishing points
F,H∞ p∞
affine
Points correspondences and internal camera calibration
w,w’ W∞
metric

9. separate known from unknown
Epipolar geometry: basic equation
separate known from unknown
(data)
(unknowns)
(linear)

10. the singularity constraint
SVD from linearly computed F matrix (rank 3)
Compute closest rank-2 approximation

12. the minimum case – 7 point correspondences
one parameter family of solutions
but F1+lF2 not automatically rank 2

13. the minimum case – impose rank 2F1 F2 F 3
F7pts
(obtain 1 or 3 solutions)
(cubic equation)
Compute possible l as eigenvalues of
(only real solutions are potential solutions)

14. ! the NOT normalized 8-point algorithm Orders of magnitude difference
~10000
~100 1 !
Orders of magnitude difference
Between column of data matrix
 least-squares yields poor results

15. Transform image to ~[-1,1]x[-1,1]
the normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)
(-1,1)
(-1,-1)
Least squares yields good results (Hartley, PAMI´97)

16. algebraic minimization
possible to iteratively minimize algebraic distance subject to det F=0 (see book if interested)

17. Geometric distance Gold standard Sampson error
Symmetric epipolar distance

18. Gold standard Maximum Likelihood Estimation
(= least-squares for Gaussian noise)
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
Parameterize:
(overparametrized)
Minimize cost using Levenberg-Marquardt
(preferably sparse LM, see book)

19. Gold standard Alternative, minimal parametrization (with a=1)
(note (x,y,1) and (x‘,y‘,1) are epipoles)
problems:
a=0
 pick largest of a,b,c,d to fix
epipole at infinity
 pick largest of x,y,w and of x’,y’,w’
4x3x3=36 parametrizations!
reparametrize at every iteration, to be sure

20. Zhang&Loop’s approach CVIU’01

21. First-order geometric error (Sampson error)
(one eq./point
JJT scalar)
(problem if some x is located at epipole)
advantage: no subsidiary variables required

22. Symmetric epipolar error

23. Some experiments:

24. Some experiments:

25. Some experiments:

26. Some experiments:
Residual error:
(for all points!)

27. Recommendations: Do not use unnormalized algorithms
Quick and easy to implement: 8-point normalized
Better: enforce rank-2 constraint during minimization
Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

28. Special case: Enforce constraints for optimal results:
Pure translation (2dof),
Planar motion (6dof),
Calibrated case (5dof)

29. The envelope of epipolar lines
What happens to an epipolar line if there is noise?
Monte Carlo
n=10
n=15
n=25
n=50

30. Other entities? Lines give no constraint for two view geometry
(but will for three and more views)
Curves and surfaces yield some constraints related to tangency

31. Automatic computation of F
Interest points
Putative correspondences
RANSAC
(iv) Non-linear re-estimation of F
Guided matching
(repeat (iv) and (v) until stable)

32. Extract feature points to relate images Required properties:
Well-defined
(i.e. neigboring points should all be different)
Stable across views
(i.e. same 3D point should be extracted
as feature for neighboring viewpoints)

33. (e.g.Harris&Stephens´88; Shi&Tomasi´94)
Feature points
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
Find points that differ as much as possible
from all neighboring points
homogeneous
edge
corner
M should have large eigenvalues
Feature = local maxima (subpixel) of F(1,  2)

34. Select strongest features (e.g. 1000/image)
Feature points
Select strongest features (e.g. 1000/image)

35. ? Evaluate NCC for all features with similar coordinates
Feature matching
Evaluate NCC for all features with
similar coordinates
Keep mutual best matches
Still many wrong matches! ?

36. Feature example 1 5 2 4 3 1 5 2 4 3 Gives satisfying results
0.96
-0.40
-0.16
-0.39
0.19
-0.05
0.75
-0.47
0.51
0.72
-0.18
0.73
0.15
-0.75
-0.27
0.49
0.16
0.79
0.21
0.08
0.50
-0.45
0.28
0.99 1 5 2 4 3
Gives satisfying results
for small image motions

37. Requirement to cope with larger variations between images
Wide-baseline matching…
Requirement to cope with larger variations between images
Translation, rotation, scaling
Foreshortening
Non-diffuse reflections
Illumination
geometric
transformations
photometric
changes

38. (Tuytelaars and Van Gool BMVC 2000)
Wide-baseline matching…
(Tuytelaars and Van Gool BMVC 2000)
Wide baseline matching for two different region types

39. RANSAC Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
(generate
hypothesis)
(verify hypothesis)
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
#inliers
90%
80%
70%
60%
50%
#samples 5 13 35
106
382

40. restrict search range to neighborhood of epipolar line
Finding more matches
restrict search range to neighborhood of epipolar line
(1.5 pixels)
relax disparity restriction (along epipolar line)

41. Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
Degenerate cases:
Degenerate cases
Planar scene
Pure rotation
No unique solution
Remaining DOF filled by noise
Use simpler model (e.g. homography)
Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
Compare H and F according to expected residual error (compensate for model complexity)

42. Absence of sufficient features (no texture)
More problems:
Absence of sufficient features (no texture)
Repeated structure ambiguity
Robust matcher also finds
support for wrong hypothesis
solution: detect repetition
(Schaffalitzky and Zisserman,
BMVC‘98)

43. geometric relations between two views is fully
two-view geometry
geometric relations between two views is fully
described by recovered 3x3 matrix F

44. Next class: image pair rectification reconstructing points and lines