1. Epipolar geometry Class 5

2. 3D photography course schedule (tentative)
Lecture
Exercise
Sept 26
Introduction -
Oct. 3
Geometry & Camera model
Camera calibration
Oct. 10
Single View Metrology
Measuring in images
Oct. 17
Feature Tracking/matching
(Friedrich Fraundorfer)
Correspondence computation
Oct. 24
Epipolar Geometry
F-matrix computation
Oct. 31
Shape-from-Silhouettes
(Li Guan)
Visual-hull computation
Nov. 7
Stereo matching
Project proposals
Nov. 14
Structured light and active range sensing
Papers
Nov. 21
Structure from motion
Nov. 28
Multi-view geometry and self-calibration
Dec. 5
Shape-from-X
Dec. 12
3D modeling and registration
Dec. 19
Appearance modeling and image-based rendering
Final project presentations

3. Optical flow Brightness constancy assumption 1D example (small motion)
It … is intensity difference (J-I)
possibility for iterative refinement

4. Optical flow Brightness constancy assumption 2D example (small motion)
the “aperture” problem
(1 constraint) ?
(2 unknowns)
isophote I(t+1)=I
isophote I(t)=I

5. Optical flow How to deal with aperture problem?
(3 constraints if color gradients are different)
Assume neighbors have same displacement

6. Lucas-Kanade
Assume neighbors have same displacement
least-squares:

7. Revisiting the small motion assumption
The back does not move a lot. But the tree in the foreground moves much more than one pixel. Not a valid first order approximation
Is this motion small enough?
Probably not—it’s much larger than one pixel (2nd order terms dominate)
How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

8. Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

9. Coarse-to-fine optical flow estimation
Gaussian pyramid of image It-1
Gaussian pyramid of image I
image I
image It-1
slides from
Bradsky and Thrun
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
image It-1
image I

10. Coarse-to-fine optical flow estimation
Gaussian pyramid of image It-1
Gaussian pyramid of image I
image I
image It-1
slides from
Bradsky and Thrun
run iterative L-K
warp & upsample
run iterative L-K .
image J
image I

11. Feature tracking Identify features and track them over video
Small difference between frames
potential large difference overall
Standard approach:
KLT (Kanade-Lukas-Tomasi)
a later image taken at time t + ¿ can be obtained by moving every point in the current image, taken
at time t, by a suitable amount. The amount of motion d = (»; ´) is called the displacement of the point at x = (x; y)
between time instants t and t + ¿ , and is in general a function of x, y, t, and ¿ .
Even in a static environment under a constant lighting, the property described by equation (2.1) is violated in many
situations. For instance, at occluding boundaries, points do not just move within the image, but appear and disappear.
Furthermore, the photometric appearance of a region on a visible surface changes when reflectivity is a function of the
viewpoint.

12. Good features to track
Use same window in feature selection as for tracking itself
Compute motion assuming it is small
Affine is also possible, but a bit harder (6x6 in stead of 2x2)
differentiate:

13. Example
Simple displacement is sufficient between consecutive frames, but not to compare to reference template

14. Example

15. Synthetic example
Affine transforms

16. Good features to keep tracking
Perform affine alignment between first and last frame
Stop tracking features with too large errors

19. Two-view geometry Three questions:
Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the corresponding point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?
(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?

20. The epipolar geometry
C,C’,x,x’ and X are coplanar

21. The epipolar geometry
What if only C,C’,x are known?

22. The epipolar geometry
All points on p project on l and l’

23. The epipolar geometry Family of planes p and lines l and l’
Intersection in e and e’

24. The epipolar geometry epipoles e,e’
= intersection of baseline with image plane
= projection of projection center in other image
= vanishing point of camera motion direction
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image
(always come in corresponding pairs)

25. Example: converging cameras

26. (simple for stereo  rectification)
Example: motion parallel with image plane
(simple for stereo  rectification)

27. Example: forward motion

28. The fundamental matrix F
algebraic representation of epipolar geometry
we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

29. The fundamental matrix F
geometric derivation
mapping from 2-D to 1-D family (rank 2)

30. The fundamental matrix F
algebraic derivation
(note: doesn’t work for C=C’  F=0)

31. The fundamental matrix F
correspondence condition
The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images

32. The fundamental matrix F
F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’
Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)
Epipolar lines: l’=Fx & l=FTx’
Epipoles: on all epipolar lines, thus e’TFx=0, x e’TF=0, similarly Fe=0
F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)
F is a correlation, projective mapping from a point x to a line l’=Fx (not a proper correlation, i.e. not invertible)

33. Fundamental matrix for pure translation

34. Fundamental matrix for pure translation

35. Fundamental matrix for pure translation
General motion
Pure translation
for pure translation F only has 2 degrees of freedom

36. The fundamental matrix F
relation to homographies
valid for all plane homographies

37. The fundamental matrix F
relation to homographies
requires

38. Projective transformation and invariance
Derivation based purely on projective concepts
F invariant to transformations of projective 3-space
unique
not unique
canonical form

39. Projective ambiguity of cameras given F
previous slide: at least projective ambiguity
this slide: not more!
Show that if F is same for (P,P’) and (P,P’),
there exists a projective transformation H so that
P=HP and P’=HP’
~ ~ ~
lemma:
(22-15=7, ok)

40. The projective reconstruction theorem
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!

41. Canonical cameras given F
Possible choice:
Canonical representation:

42. Fundamental matrix (3x3 rank 2 matrix)
Epipolar geometry l2 C1 m1 L1 m2 L2 M C2 C1 C2 l2 p l1 e1 e2 m1 L1 m2 L2 M C1 C2 l2 p l1 e1 e2
Underlying structure in set of matches for rigid scenes m1 m2
lT1 l2
Fundamental matrix (3x3 rank 2 matrix)
Computable from corresponding points
Simplifies matching
Allows to detect wrong matches
Related to calibration
Canonical representation:

43. Epipolar geometry?
courtesy Frank Dellaert

44. Other entities besides points?
Lines give no constraint for two view geometry
(but will for three and more views)
Curves and surfaces yield some constraints related to tangency
(e.g. Sinha et al. CVPR’04)

45. Computation of F Linear (8-point) Minimal (7-point) Robust (RANSAC)
Non-linear refinement (MLE, …)
Practical approach

46. separate known from unknown
Epipolar geometry: basic equation
separate known from unknown
(data)
(unknowns)
(linear)

47. ! the NOT normalized 8-point algorithm Orders of magnitude difference
~10000
~100 1 !
Orders of magnitude difference
between column of data matrix
 least-squares yields poor results

48. Transform image to ~[-1,1]x[-1,1]
the normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)
(-1,1)
(-1,-1)
normalized least squares yields good results (Hartley, PAMI´97)

49. the singularity constraint
SVD from linearly computed F matrix (rank 3)
Compute closest rank-2 approximation

51. the minimum case – 7 point correspondences
one parameter family of solutions
but F1+lF2 not automatically rank 2

52. the minimum case – impose rank 2F1 F2 F 3
F7pts
(obtain 1 or 3 solutions)
(cubic equation)
Compute possible l as eigenvalues of
(only real solutions are potential solutions)

53. Automatic computation of F
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
(generate
hypothesis)
(verify hypothesis)
RANSAC
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
#inliers
90%
80%
70%
60%
50%
#samples 5 13 35
106
382

54. restrict search range to neighborhood of epipolar line
Finding more matches
restrict search range to neighborhood of epipolar line
(e.g. 1.5 pixels)
relax disparity restriction (along epipolar line)

55. (Mostly) planar scene (see next slide)
Issues:
(Mostly) planar scene (see next slide)
Absence of sufficient features (no texture)
Repeated structure ambiguity
Robust matcher also finds
support for wrong hypothesis
solution: detect repetition
(Schaffalitzky and Zisserman,
BMVC‘98)

56. Computing F for quasi-planar scenes QDEGSAC
337 matches on plane, 11 off plane
#inliers
%inclusion of
out-of-plane inliers
17% success for RANSAC
100% for QDEGSAC
data rank

57. geometric relations between two views is fully
two-view geometry
geometric relations between two views is fully
described by recovered 3x3 matrix F