Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2 Tutorial outline Introduction Visual 3D modeling –Acquisition of camera motion –Acquisition of scene structure –Constructing visual models Examples and applications

3 Visual 3D models from images Visualization –Virtual/augmented/mixed reality, tele-presence, medicine, simulation, e-commerce, etc. Metrology –Cultural heritage and archaeology, geology, forensics, robot navigation, etc. “Sampling” the real world Convergence of computer vision, graphics and photogrammetry

4 Visual 3D models from video Scene (static) Visual model camera unknown scene unknown camera unknown motion automatic modelling What can be achieved?

5 Example: DV video  3D model accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)

6 (Pollefeys et al. ICCV’98; … Pollefeys et al.’IJCV04)

7 Outline Introduction Image formation Relating multiple views

8 Perspective projection Linear equations (in homogeneous coordinates)

9 Homogeneous coordinates 2-D points represented as 3-D vectors (x y 1) T 3-D points represented as 4-D vectors (X Y Z 1) T Equality defined up to scale –(X Y Z 1) T ~ (WX WY WZ W) T Useful for perspective projection  makes equations linear C m M1M1 M2M2

Pinhole camera model linear projection in homogeneous coordinates! 10

11 The pinhole camera

12 Effects of perspective projection Colinearity is invariant Parallelism is not preserved

Principal point offset principal point 13

Principal point offset calibration matrix 14

Camera rotation and translation 15

16 Intrinsic parameters Camera deviates from pinhole s: skew f x ≠ f y : different magnification in x and y (c x c y ): optical axis does not pierce image plane exactly at the center Usually: rectangular pixels: square pixels: principal point known: or

17 Extrinsic parameters Scene motion Camera motion

18 Projection matrix Includes coordinate transformation and camera intrinsic parameters

19 Projection matrix Mapping from 2-D to 3-D is a function of internal and external parameters

20 Radial distortion In reality, straight lines are not preserved due to lens distortion Estimate polynomial model to correct it

21 Radial distortion

22 Outline Introduction Image formation Relating multiple views

23 l2l2 3D from images C1C1 m1m1 M? L1L1 m2m2 L2L2 M C2C2 Triangulation - calibration - correspondences

24 Compare intensities pixel-by-pixel Comparing image regions I(x,y) I´(x,y) Normalized Cross Correlation Sum of Square Differences (Dis)similarity measures

25 Structure and motion recovery Self-calibration Feature extraction Feature matching Multi-view relation Structure and motion recovery

26 Other cues for depth and geometry Shading Shadows, symmetry, silhouette Texture Focus

27 Epipolar geometry C1C1 C2C2 l2l2 P l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 m1m1 m2m2 C1C1 C2C2 l2l2 P l1l1 e1e1 e2e2 m1m1 L1L1 m2m2 L2L2 M l2l2 lT1lT1

28 The epipoles Image 1 Image 2 C1C1 e 12 C2C2 e 21 The epipole is the projection of the focal point of one camera in another image.

separate known from unknown (data) (unknowns) (linear) 29 Two view geometry computation: linear algorithm For every match (m,m´):

30 Benefits from having F Given a pixel in one image, the corresponding pixel has to lie on epipolar line Search space reduced from 2-D to 1-D

31 Two view geometry computation: finding more matches

32 Difficulties in finding corresponding pixels

33 Matching difficulties Occlusion Absence of sufficient features (no texture) Smoothness vs. sensitivity Double nail illusion

34 Two view geometry computation: more problems Repeated structure ambiguity Robust matcher also finds support for wrong hypothesis solution: detect repetition