Multiple View Geometry Marc Pollefeys COMP 256

Last class Gaussian pyramid Laplacian pyramid Gabor Fourier filters transform

Histograms : co-occurrence matrix Not last class … Histograms : co-occurrence matrix

Texture synthesis [Zalesny & Van Gool 2000] 2 analysis iterations 6 analysis iterations 9 analysis iterations

View-dependent texture synthesis [Zalesny & Van Gool 2000]

Efros & Leung ’99 p Assuming Markov property, compute P(p|N(p)) non-parametric sampling Synthesizing a pixel Input image Assuming Markov property, compute P(p|N(p)) Building explicit probability tables infeasible Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p To synthesize p, just pick one match at random

Efros & Leung ’99 extended p B Idea: unit of synthesis = block Exactly the same but now we want P(B|N(B)) Much faster: synthesize all pixels in a block at once Not the same as multi-scale! Synthesizing a block non-parametric sampling Input image Observation: neighbor pixels are highly correlated

constrained by overlap block Input texture B1 B2 Neighboring blocks constrained by overlap B1 B2 Minimal error boundary cut B1 B2 Random placement of blocks

Minimal error boundary overlapping blocks vertical boundary _ = 2 overlap error min. error boundary

Why do we see more flowers in the distance? [Leung & Malik CVPR97] Perpendicular textures

Shape-from-texture

Tentative class schedule Jan 16/18 - Introduction Jan 23/25 Cameras Radiometry Jan 30/Feb1 Sources & Shadows Color Feb 6/8 Linear filters & edges Texture Feb 13/15 Multi-View Geometry Stereo Feb 20/22 Optical flow Project proposals Feb27/Mar1 Affine SfM Projective SfM Mar 6/8 Camera Calibration Silhouettes and Photoconsistency Mar 13/15 Springbreak Mar 20/22 Segmentation Fitting Mar 27/29 Prob. Segmentation Project Update Apr 3/5 Tracking Apr 10/12 Object Recognition Apr 17/19 Range data Apr 24/26 Final project

THE GEOMETRY OF MULTIPLE VIEWS Epipolar Geometry The Essential Matrix The Fundamental Matrix The Trifocal Tensor The Quadrifocal Tensor Reading: Chapter 10.

Epipolar Geometry Epipolar Plane Baseline Epipoles Epipolar Lines

Potential matches for p have to lie on the corresponding Epipolar Constraint Potential matches for p have to lie on the corresponding epipolar line l’. Potential matches for p’ have to lie on the corresponding epipolar line l.

Epipolar Constraint: Calibrated Case Essential Matrix (Longuet-Higgins, 1981)

E p’ is the epipolar line associated with p’. Properties of the Essential Matrix T E p’ is the epipolar line associated with p’. ETp is the epipolar line associated with p. E e’=0 and ETe=0. E is singular. E has two equal non-zero singular values (Huang and Faugeras, 1989). T

Epipolar Constraint: Small Motions To First-Order: Pure translation: Focus of Expansion

Epipolar Constraint: Uncalibrated Case Fundamental Matrix (Faugeras and Luong, 1992)

Properties of the Fundamental Matrix F p’ is the epipolar line associated with p’. FT p is the epipolar line associated with p. F e’=0 and FT e=0. F is singular. T T

The Eight-Point Algorithm (Longuet-Higgins, 1981) |F| =1. Minimize: under the constraint 2

Non-Linear Least-Squares Approach (Luong et al., 1993) Minimize with respect to the coefficients of F , using an appropriate rank-2 parameterization.

Problem with eight-point algorithm linear least-squares: unit norm vector F yielding smallest residual What happens when there is noise?

The Normalized Eight-Point Algorithm (Hartley, 1995) Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’. Use the eight-point algorithm to compute F from the points q and q’ . Enforce the rank-2 constraint. Output T F T’. i i i i i i T

Epipolar geometry example

courtesy of Andrew Zisserman Example: converging cameras courtesy of Andrew Zisserman

Example: motion parallel with image plane (simple for stereo  rectification) courtesy of Andrew Zisserman

courtesy of Andrew Zisserman Example: forward motion e’ e courtesy of Andrew Zisserman

courtesy of Andrew Zisserman Fundamental matrix for pure translation auto-epipolar courtesy of Andrew Zisserman

courtesy of Andrew Zisserman Fundamental matrix for pure translation courtesy of Andrew Zisserman

Trinocular Epipolar Constraints These constraints are not independent!

Trinocular Epipolar Constraints: Transfer Given p and p , p can be computed as the solution of linear equations. 1 2 3

Trinocular Epipolar Constraints: Transfer problem for epipolar transfer in trifocal plane! There must be more to trifocal geometry… image from Hartley and Zisserman

Trifocal Constraints

Trifocal Constraints Calibrated Case All 3x3 minors must be zero! Trifocal Tensor

Trifocal Constraints Uncalibrated Case Trifocal Tensor

Trifocal Constraints: 3 Points Pick any two lines l and l through p and p . 2 3 2 3 Do it again. T( p , p , p )=0 1 2 3

For any matching epipolar lines, l G l = 0. Properties of the Trifocal Tensor For any matching epipolar lines, l G l = 0. The matrices G are singular. They satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995). T i 2 1 3 i 1 Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-squares Impose the constraints a posteriori.

For any matching epipolar lines, l G l = 0. 2 1 3 The backprojections of the two lines do not define a line!

courtesy of Andrew Zisserman Trifocal Tensor Example 108 putative matches 18 outliers (26 samples) 88 inliers 95 final inliers (0.43) (0.23) (0.19) courtesy of Andrew Zisserman

Trifocal Tensor Example additional line matches images courtesy of Andrew Zisserman

Transfer: trifocal transfer (using tensor notation) doesn’t work if l’=epipolar line image courtesy of Hartley and Zisserman

Image warping using T(1,2,N) (Avidan and Shashua `97)

Multiple Views (Faugeras and Mourrain, 1995)

Two Views Epipolar Constraint

Three Views Trifocal Constraint

Four Views Quadrifocal Constraint (Triggs, 1995)

Geometrically, the four rays must intersect in P..

Quadrifocal Tensor and Lines

Quadrifocal tensor determinant is multilinear thus linear in coefficients of lines ! There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor

Scale-Restraint Condition from Photogrammetry

from perspective to omnidirectional cameras 3 constraints allow to reconstruct 3D point perspective camera (2 constraints / feature) more constraints also tell something about cameras l=(y,-x) (x,y) (0,0) multilinear constraints known as epipolar, trifocal and quadrifocal constraints radial camera (uncalibrated) (1 constraints / feature)

Radial quadrifocal tensor (x,y) Radial quadrifocal tensor Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views Reconstruct 3D scene and use it for calibration (2x2x2x2 tensor) Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation Radial trifocal tensor Tijk from 7 points in 3 views Reconstruct 2D panorama and use it for calibration (2x2x2 tensor)

Non-parametric distortion calibration (Thirthala and Pollefeys, ICCV’05) Models fish-eye lenses, cata-dioptric systems, etc. angle normalized radius

Non-parametric distortion calibration (Thirthala and Pollefeys, ICCV’05) Models fish-eye lenses, cata-dioptric systems, etc. 90o angle normalized radius

Next class: Stereo (x´,y´)=(x+D(x,y),y) F&P Chapter 11 image I´(x´,y´) Disparity map D(x,y) image I´(x´,y´) (x´,y´)=(x+D(x,y),y) F&P Chapter 11