1. MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling

2. MASKS © 2004 Invitation to 3D vision Review Feature correspondence Camera Calibration Feature selection Feature selection Euclidean Reconstruction Sparse Structure and camera motion Landing Augmented Reality Vision Based Control

3. MASKS © 2004 Invitation to 3D vision Review Feature correspondence Camera Calibration Dense Correspondence Texture mapping Epipolar Rectification Feature selection Feature selection 3-D Model Sparse Structure and motion Euclidean Reconstruction

4. MASKS © 2004 Invitation to 3D vision Review Feature correspondence Projective Reconstruction Euclidean Reconstruction Dense Correspondence Texture mapping Epipolar Rectification Feature selection Feature selection 3-D Model Camera Self-Calibration Partial Scene Knowledge Partial Motion Knowledge Partial Calibration Knowledge

5. MASKS © 2004 Invitation to 3D vision Examples

6. MASKS © 2004 Invitation to 3D vision Feature Selection Compute Image Gradient Compute Feature Quality measure for each pixel Search for local maxima Feature Quality FunctionLocal maxima of feature quality function

7. MASKS © 2004 Invitation to 3D vision Feature Tracking Translational motion model Closed form solution 1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level

8. MASKS © 2004 Invitation to 3D vision Coarse to fine feature tracking 1. compute 2. warp the window in the second image by 3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations 0 2 1

9. MASKS © 2004 Invitation to 3D vision Integrate around over image patch Solve Optical Flow

10. MASKS © 2004 Invitation to 3D vision Affine feature tracking Intensity offset Contrast change

11. MASKS © 2004 Invitation to 3D vision Tracked Features

12. MASKS © 2004 Invitation to 3D vision Wide baseline matching Point features detected by Harris Corner detector

13. MASKS © 2004 Invitation to 3D vision Wide baseline Feature Matching 1. Select the features in two views 2. For each feature in the first view 3. Find the feature in the second view that maximizes 4. Normalized cross-correlation measure Select the candidate with the similarity above selected threshold

14. MASKS © 2004 Invitation to 3D vision More correspondences and Robust matching Select set of putative correspondences Repeat 1. Select at random a set of 8 successful matches 2. Compute fundamental matrix 3. Determine the subset of inliers, compute distance to epipolar line 4. Count the number of points in the consensus set

15. MASKS © 2004 Invitation to 3D vision RANSAC in action Inliers Outliers

16. MASKS © 2004 Invitation to 3D vision Epipolar Geometry Epipolar geometry in two views Refined epipolar geometry using nonlinear estimation of F

17. MASKS © 2004 Invitation to 3D vision Two view initialization Recover epipolar geometry Compute (Euclidean) projection matrices and 3-D struct. Compute (Projective) projection matrices and 3-D struct. calibrated uncalibrated unknown

18. MASKS © 2004 Invitation to 3D vision Projective Reconstruction 3-D reconstruction from two views

19. MASKS © 2004 Invitation to 3D vision Multi-view reconstruction Two view - initialized motion and structure estimates (scales) Multi-view factorization - recover the remaining camera positions and refine the 3-D structure by iteratively computing 1. Compute i-th motion given the known structure 2. Refine structure estimate given all the motions Repeat until convergence iteration

20. MASKS © 2004 Invitation to 3D vision Projective Ambiguity calibrated case – projection matrices - and Euclidean structure Uncalibrated case for i-th frame for 1 st frame Transform projection matrices and 3-D structure by H

21. MASKS © 2004 Invitation to 3D vision From Projective to Euclidean Reconstruction Remove the ambiguity characterized by How to estimate H ? - Absolute quadric constraint be recovered by a nonlinear minimization unknowns Upgrade to Euclidean Reconstruction

22. MASKS © 2004 Invitation to 3D vision Upgrade to Euclidean Reconstruction Special case unknown focal length other internal parameters are known (proj. center, aspect ratio) Q can be parametrized in the following way Zero entries in lead to linear constraints on Q Initial estimate of Q can be obtained using linear techniques

23. MASKS © 2004 Invitation to 3D vision Example of multi-view reconstruction Euclidean reconstruction Projective Reconstruction

24. MASKS © 2004 Invitation to 3D vision Nonlinear Refinement Euclidean Bundle adjustment Initial estimates of are available Final refinement, nonlinear minimization with respect to all unknowns

25. MASKS © 2004 Invitation to 3D vision Example - Euclidean multi-view reconstruction

26. MASKS © 2004 Invitation to 3D vision Example Tracked Features Original sequence

27. MASKS © 2004 Invitation to 3D vision Recovered model

28. MASKS © 2004 Invitation to 3D vision Euclidean Reconstruction

29. MASKS © 2004 Invitation to 3D vision Epipolar rectification Make the epipolar lines parallel Dense correspondences along image scanlines Computation of warping homographies 1. Map the epipole to infinity Translate the image center to the origin Rotate around z-axis for the epipole lie on the x-axis Transform the epipole from x-axis to infinity 2. Find a matching transformation is compatible with the epipolar geometry is chosen to minimize overall disparity

30. MASKS © 2004 Invitation to 3D vision Epipolar rectification Rectified Image Pair

31. MASKS © 2004 Invitation to 3D vision Epipolar rectification Rectified Image Pair

32. MASKS © 2004 Invitation to 3D vision Dense Matching Establish dense correspondences along scan-lines Standard stereo configuration Constraints to guide the search 1. ordering constraint 2. disparity constraint – limit on disparity 3. uniqueness constraint – each point has a unique match in the second view

33. MASKS © 2004 Invitation to 3D vision Dense Matching

34. MASKS © 2004 Invitation to 3D vision Dense Reconstruction

35. MASKS © 2004 Invitation to 3D vision Texture mapping, hole filling

36. MASKS © 2004 Invitation to 3D vision Texture mapping