1. EECS 274 Computer Vision
Stereopsis

2. Stereopsis Epipolar geometry Trifocal tensor
Stereopsis: Fusion and Reconstruction
Correlation-Based Fusion
Multi-Scale Edge Matching
Dynamic Programming
Using Three or More Cameras
Reading: FP Chapter 7, S Chapter 11

3. Mobile Robot Navigation
The INRIA Mobile Robot, 1990.
The Stanford Cart,
H. Moravec, 1979.
Courtesy O. Faugeras and H. Moravec.

4. Stereo vision Fusion: match points observed by two or more cameras
Reconstruction: find the pre-image of the matching points in 3D world
Assume calibrated camera and essential matrix or trifocal tensors associated with 3 cameras are known

5. Reconstruction/triangulizaiton

6. Binocular fusion

7. Epipolar geometry Epipolar plane defined by P, O, O’, p and p’
Baseline OO’
p’ lies on l’ where the epipolar plane intersects with image plane π’
l’ is epipolar line associated with p and intersects baseline OO’ on e’
e’ is the projection of O observed from O’
Epipolar lines l, l’
Epipoles e, e’

8. Epipolar constraint
Potential matches for p have to lie on the corresponding
epipolar line l’
Potential matches for p’ have to lie on the corresponding
epipolar line l

9. Epipolar Constraint: Calibrated Case
M’=(R t)
Essential Matrix
(Longuet-Higgins, 1981)
3 ×3 skew-symmetric matrix: rank=2

10. Properties of essential matrix
E is defined by 5 parameters (3 for rotation and 2 for translation)
E p’ is the epipolar line associated with p’
E T p is the epipolar line associated with p
Can write as l .p = 0
The point p lies on the epipolar line associated with the vector
E p’

11. Properties of essential matrix (cont’d)
E e’=0 and ETe=0 (E e’=-RT[tx]e=0 )
E is singular
E has two equal non-zero singular values (Huang and Faugeras, 1989)

12. Epipolar Constraint: Small Motions To First-Order:
Longuet-Higgins relation
Pure translation: Focus of Expansion e
The motion field at every point in
the image points to focus of expansion

13. Epipolar Constraint: Uncalibrated Case
Fundamental Matrix
(Faugeras and Luong, 1992)
are normalized image coordinate
K, K’: calibration matrices

14. Properties of fundamental matrix
F has rank 2 and is defined by 7 parameters
F p’ is the epipolar line associated with p’ in the 1st image
F T p is the epipolar line associated with p in the 2nd image
F e’=0 and F T e=0
F is singular

15. Rank-2 constraint F admits 7 independent parameter
Possible choice of parameterization using e=(α,β)T and e’=(α’,β’)T and epipolar transformation
Can be written with 4 parameters: a, b, c, d

16. Weak calibration In theory:
E can be estimated with 5 point correspondences
F can be estimated with 7 point correspondences
Some methods estimate E and F matrices from a minimal number of parameters
Estimating epipolar geometry from a redundant set of point correspondences with unknown intrinsic parameters

17. The Eight-Point Algorithm (Longuet-Higgins, 1981)
Homogenous system, set F33=1
Use 8 point correspondences
|F | =1.
Minimize:
under the constraint 2

18. Least-squares minimization
|F | =1
Minimize:
under the constraint
Error function:
d(p,l): Euclidean distance between point p and line l

19. Non-Linear Least-Squares Approach (Luong et al., 1993)
8 point algorithm with least-squares minimization
ignores the rank 2 property
First use least squares to find epipoles e and e’ that
minimizes |FT e|2 and |Fe’|2
Minimize
with respect to the coefficients of F , using an
appropriate rank-2 parameterization (4 parameters
instead of 8)

20. The Normalized Eight-Point Algorithm (Hartley, 1995)
Estimation of transformation parameters suffer form poor numerical condition problem
Center the image data at the origin, and scale it so the
mean squared distance between the origin and the data
points is 2 pixels: q = T p , q’ = T’ p’
Use the eight-point algorithm to compute F from the
points q and q’
Enforce the rank-2 constraint
Output T F T’ i i i i i T

21. Weak calibration experiment

22. Trinocular Epipolar Constraints
Optical centers O1O2O3 defines a trifocal plane
Generally, P does not lie on trifocal plane formed
Trifocal plane intersects retinas along t1, t2, t3
Each line defines two epipoles, e.g., t2 defines e12, e32, wrt O1 and O3
These constraints are not independent!

23. Trinocular Epipolar Constraints: Transfer
Given p1 and p2 , p3 can be computed
as the solution of linear equations.
Geometrically, p1 is found as the intersection
of epipolar lines associated with p2 and p3

24. Trifocal Constraints P
The set of points that project onto an image line l is the plane L that contains the line and pinhole
Point P in L is projected onto p on line l (l=(a,b,c)T)
Recall

25. Trifocal Constraints Calibrated Case All 3×3 minors must be zero!P
Calibrated Case
All 3×3 minors
must be zero!
line-line-line correspondence
Trifocal Tensor

26. Trifocal Constraints Calibrated Case
Given 3 point correspondences, p1, p2, p3 of the same point P, and two lines l2, l3, (passing through p2, and p3), O1p1 must intersect the line l, where the planes L2 and L3 intersect
point-line-line correspondence

27. Trifocal Constraints P
Uncalibrated Case

28. Trifocal Constraints
Uncalibrated Case
Trifocal Tensor

29. Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p . 2 3 2 3
Do it again.
T( p , p , p )=0 1 2 3

30. Properties of the Trifocal Tensor
For any matching epipolar lines, l G l = 0
The matrices G are singular
Each triple of points  4 independent equations
Each triple of lines  2 independent equations
4p+2l ≥ 26  need 7 triples of points or 13 triples of lines
The coefficients of tensor satisfy 8 independent constraints
in the uncalibrated case (Faugeras and Mourrain, 1995)
 Reduce the number of independent parameters from 26 to 18 2 1 3 T i i 1
Estimating the Trifocal Tensor
Ignore the non-linear constraints and use linear least-squares
a posteriori
Impose the constraints a posteriori

31. Multiple Views (Faugeras and Mourrain, 1995)
All 4 × 4 minors
have zero
determinants

32. Two Views
6 minors
Epipolar Constraint

33. Three Views
48 minors
Trifocal Constraint

34. Quadrifocal Constraint (Triggs, 1995)
Four Views
16 minors
Quadrifocal Constraint
(Triggs, 1995)

35. Geometrically, the four rays must intersect in P..

36. Quadrifocal Tensor and Lines
Given 4 point correspondences, p1, p2, p3, p4 of the same point P, and 3 lines l2, l3, l4 (passing through p2, and p3, p4), O1p1 must intersect the line l, where the planes L2 , L3, and L4

37. Scale-Restraint Condition from Photogrammetry
Trinocular constraints in the presence of calibration or measurement errors

38. Stereo: Reconstruction
Geometric
approach
Algebraic approach
Linear Method:
find P such that
(least squares)
Non-Linear Method: find Q minimizing

39. Retification
All epipolar lines are parallel in the rectified image plane

40. Reconstruction from rectified images
Disparity: d=u’-u
B: baseline between O and O’
Depth: z = -B/d × f

41. Correlation methods (1970--)
Slide the window along the epipolar line until w.w’ is maximized. 2
Minimize ||w-w’||
Normalized Correlation: minimize q instead.

42. Forshortening problems
Solution: add a second pass using disparity estimates to warp
the correlation windows, e.g. Devernay and Faugeras (1994)

43. Multiscale edge matching
[Marr, Poggio and Grimson, ]
Edges are found by repeatedly smoothing the image and detecting
the zero crossings of the second derivative (Laplacian)
Matches at coarse scales are used to offset the search for matches
at fine scales (equivalent to eye movements)

44. Multiscale edge matching
One of the two
input images
Image Laplacian
Zeros of the Laplacian

45. Multiscale edge matching

46. Ordering constraints In general the points are in the same order
on both epipolar lines.
But it is not always the case..

47. Ordering constraints points are not necessarily in order
d-b-a c’-b’-d’

48. Dynamic programming Find the minimum-cost path going monotonically
[Baker and Binford, 1981]
Find the minimum-cost path going monotonically
down and right from the top-left corner of the
graph to its bottom-right corner.
Nodes = matched feature points (e.g., edge points).
Arcs = matched intervals along the epipolar lines.
Arc cost = discrepancy between intervals.

49. Dynamic programming [Ohta and Kanade, 1985]
use inter-scanline f or point
correspondence on vertical
edges
[Ohta and Kanade, 1985]
use DP for both
intra-scanline and
inter-scanline search

50. Three views The third eye can be used for verification..
b1-a2 match is wrong as thee is no corresponding point in camera 3

51. More views Pick a reference image, and slide the corresponding
window along the corresponding epipolar lines of all
other images, using inverse depth relative to the first
image as the search parameter.
[Okutami and Kanade, 1993]
Use the sum of correlation scores to rank matches.

52. I1 I2
I10

53. Correspondence Extensive literature Smooth surfaces Region based
Graph cut
Mutual information
Belief propagation
Conditional random field
Smooth surfaces

54. Middlebury stereo dataset
De facto data set with ground truth, code, and comprehensive performance evaluation

55. Performance evaluation