EECS 274 Computer Vision Stereopsis.

Slides:



Advertisements
Similar presentations
Epipolar Geometry.
Advertisements

The fundamental matrix F
Lecture 11: Two-view geometry
3D reconstruction.
MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
Jan-Michael Frahm, Enrique Dunn Spring 2012
Two-view geometry.
Lecture 8: Stereo.
Camera calibration and epipolar geometry
Structure from motion.
Stereo. STEREOPSIS Reading: Chapter 11. The Stereopsis Problem: Fusion and Reconstruction Human Stereopsis and Random Dot Stereograms Cooperative Algorithms.
Stereo and Epipolar geometry
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
Uncalibrated Geometry & Stratification Sastry and Yang
Lecture 21: Multiple-view geometry and structure from motion
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Multiple View Geometry
CS 223b 1 More on stereo and correspondence. CS 223b 2 =?f g Mostpopular For each window, match to closest window on epipolar line in other image. (slides.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Lecture 20: Two-view geometry CS6670: Computer Vision Noah Snavely.
May 2004Stereo1 Introduction to Computer Vision CS / ECE 181B Tuesday, May 11, 2004  Multiple view geometry and stereo  Handout #6 available (check with.
3D Vision and Graphics. The Geometry of Multiple Views Motion Field Stereo Epipolar Geometry The Essential Matrix The Fundamental Matrix Structure from.
Lec 21: Fundamental Matrix
CSE473/573 – Stereo Correspondence
Multiple View Geometry
Multiple View Geometry. THE GEOMETRY OF MULTIPLE VIEWS Reading: Chapter 10. Epipolar Geometry The Essential Matrix The Fundamental Matrix The Trifocal.
CS 558 C OMPUTER V ISION Lecture IX: Dimensionality Reduction.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
Automatic Camera Calibration
Epipolar Geometry and Stereo Vision Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 03/05/15 Many slides adapted from Lana Lazebnik,
Computer vision: models, learning and inference
Multi-view geometry.
Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T Computer Vision Tahun: 2010.
Epipolar geometry The fundamental matrix and the tensor
1 Preview At least two views are required to access the depth of a scene point and in turn to reconstruct scene structure Multiple views can be obtained.
Projective cameras Motivation Elements of Projective Geometry Projective structure from motion Planches : –
Lecture 04 22/11/2011 Shai Avidan הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:
Binocular Stereo #1. Topics 1. Principle 2. binocular stereo basic equation 3. epipolar line 4. features and strategies for matching.
Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.
Announcements Project 3 due Thursday by 11:59pm Demos on Friday; signup on CMS Prelim to be distributed in class Friday, due Wednesday by the beginning.
Geometry of Multiple Views
Computer Vision Lecture #10 Hossam Abdelmunim 1 & Aly A. Farag 2 1 Computer & Systems Engineering Department, Ain Shams University, Cairo, Egypt 2 Electerical.
Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform  Location of the impulse.
Raquel A. Romano 1 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Geometry for Computer Vision Raquel A.
Two-view geometry. Epipolar Plane – plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the.
Computer vision: models, learning and inference M Ahad Multiple Cameras
Correspondence and Stereopsis Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri]
Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : –
EECS 274 Computer Vision Projective Structure from Motion.
Multiple View Geometry and Stereo. Overview Single camera geometry – Recap of Homogenous coordinates – Perspective projection model – Camera calibration.
Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.
Correspondence and Stereopsis. Introduction Disparity – Informally: difference between two pictures – Allows us to gain a strong sense of depth Stereopsis.
CSE 185 Introduction to Computer Vision Stereo 2.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.
STEREOPSIS The Stereopsis Problem: Fusion and Reconstruction
STEREOPSIS The Stereopsis Problem: Fusion and Reconstruction
Two-view geometry Computer Vision Spring 2018, Lecture 10
Epipolar geometry.
EECS 274 Computer Vision Stereopsis.
Reconstruction.
Two-view geometry.
Two-view geometry.
Multi-view geometry.
Chapter 11: Stereopsis Stereopsis: Fusing the pictures taken by two cameras and exploiting the difference (or disparity) between them to obtain the depth.
Presentation transcript:

EECS 274 Computer Vision Stereopsis

Stereopsis Epipolar geometry Trifocal tensor Stereopsis: Fusion and Reconstruction Correlation-Based Fusion Multi-Scale Edge Matching Dynamic Programming Using Three or More Cameras Reading: FP Chapter 7, S Chapter 11

Mobile Robot Navigation The INRIA Mobile Robot, 1990. The Stanford Cart, H. Moravec, 1979. Courtesy O. Faugeras and H. Moravec.

Stereo vision Fusion: match points observed by two or more cameras Reconstruction: find the pre-image of the matching points in 3D world Assume calibrated camera and essential matrix or trifocal tensors associated with 3 cameras are known

Reconstruction/triangulizaiton

Binocular fusion

Epipolar geometry Epipolar plane defined by P, O, O’, p and p’ Baseline OO’ p’ lies on l’ where the epipolar plane intersects with image plane π’ l’ is epipolar line associated with p and intersects baseline OO’ on e’ e’ is the projection of O observed from O’ Epipolar lines l, l’ Epipoles e, e’

Epipolar constraint Potential matches for p have to lie on the corresponding epipolar line l’ Potential matches for p’ have to lie on the corresponding epipolar line l

Epipolar Constraint: Calibrated Case M’=(R t) Essential Matrix (Longuet-Higgins, 1981) 3 ×3 skew-symmetric matrix: rank=2

Properties of essential matrix E is defined by 5 parameters (3 for rotation and 2 for translation) E p’ is the epipolar line associated with p’ E T p is the epipolar line associated with p Can write as l .p = 0 The point p lies on the epipolar line associated with the vector E p’

Properties of essential matrix (cont’d) E e’=0 and ETe=0 (E e’=-RT[tx]e=0 ) E is singular E has two equal non-zero singular values (Huang and Faugeras, 1989)

Epipolar Constraint: Small Motions To First-Order: Longuet-Higgins relation Pure translation: Focus of Expansion e The motion field at every point in the image points to focus of expansion

Epipolar Constraint: Uncalibrated Case Fundamental Matrix (Faugeras and Luong, 1992) are normalized image coordinate K, K’: calibration matrices

Properties of fundamental matrix F has rank 2 and is defined by 7 parameters F p’ is the epipolar line associated with p’ in the 1st image F T p is the epipolar line associated with p in the 2nd image F e’=0 and F T e=0 F is singular

Rank-2 constraint F admits 7 independent parameter Possible choice of parameterization using e=(α,β)T and e’=(α’,β’)T and epipolar transformation Can be written with 4 parameters: a, b, c, d

Weak calibration In theory: E can be estimated with 5 point correspondences F can be estimated with 7 point correspondences Some methods estimate E and F matrices from a minimal number of parameters Estimating epipolar geometry from a redundant set of point correspondences with unknown intrinsic parameters

The Eight-Point Algorithm (Longuet-Higgins, 1981) Homogenous system, set F33=1 Use 8 point correspondences |F | =1. Minimize: under the constraint 2

Least-squares minimization |F | =1 Minimize: under the constraint Error function: d(p,l): Euclidean distance between point p and line l

Non-Linear Least-Squares Approach (Luong et al., 1993) 8 point algorithm with least-squares minimization ignores the rank 2 property First use least squares to find epipoles e and e’ that minimizes |FT e|2 and |Fe’|2 Minimize with respect to the coefficients of F , using an appropriate rank-2 parameterization (4 parameters instead of 8)

The Normalized Eight-Point Algorithm (Hartley, 1995) Estimation of transformation parameters suffer form poor numerical condition problem Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’ Use the eight-point algorithm to compute F from the points q and q’ Enforce the rank-2 constraint Output T F T’ i i i i i T

Weak calibration experiment

Trinocular Epipolar Constraints Optical centers O1O2O3 defines a trifocal plane Generally, P does not lie on trifocal plane formed Trifocal plane intersects retinas along t1, t2, t3 Each line defines two epipoles, e.g., t2 defines e12, e32, wrt O1 and O3 These constraints are not independent!

Trinocular Epipolar Constraints: Transfer Given p1 and p2 , p3 can be computed as the solution of linear equations. Geometrically, p1 is found as the intersection of epipolar lines associated with p2 and p3

Trifocal Constraints P The set of points that project onto an image line l is the plane L that contains the line and pinhole Point P in L is projected onto p on line l (l=(a,b,c)T) Recall

Trifocal Constraints Calibrated Case All 3×3 minors must be zero! P Calibrated Case All 3×3 minors must be zero! line-line-line correspondence Trifocal Tensor

Trifocal Constraints Calibrated Case Given 3 point correspondences, p1, p2, p3 of the same point P, and two lines l2, l3, (passing through p2, and p3), O1p1 must intersect the line l, where the planes L2 and L3 intersect point-line-line correspondence

Trifocal Constraints P Uncalibrated Case

Trifocal Constraints Uncalibrated Case Trifocal Tensor

Trifocal Constraints: 3 Points Pick any two lines l and l through p and p . 2 3 2 3 Do it again. T( p , p , p )=0 1 2 3

Properties of the Trifocal Tensor For any matching epipolar lines, l G l = 0 The matrices G are singular Each triple of points  4 independent equations Each triple of lines  2 independent equations 4p+2l ≥ 26  need 7 triples of points or 13 triples of lines The coefficients of tensor satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995)  Reduce the number of independent parameters from 26 to 18 2 1 3 T i i 1 Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-squares a posteriori Impose the constraints a posteriori

Multiple Views (Faugeras and Mourrain, 1995) All 4 × 4 minors have zero determinants

Two Views 6 minors Epipolar Constraint

Three Views 48 minors Trifocal Constraint

Quadrifocal Constraint (Triggs, 1995) Four Views 16 minors Quadrifocal Constraint (Triggs, 1995)

Geometrically, the four rays must intersect in P..

Quadrifocal Tensor and Lines Given 4 point correspondences, p1, p2, p3, p4 of the same point P, and 3 lines l2, l3, l4 (passing through p2, and p3, p4), O1p1 must intersect the line l, where the planes L2 , L3, and L4

Scale-Restraint Condition from Photogrammetry Trinocular constraints in the presence of calibration or measurement errors

Stereo: Reconstruction Geometric approach   Algebraic approach Linear Method: find P such that (least squares) Non-Linear Method: find Q minimizing

Retification All epipolar lines are parallel in the rectified image plane

Reconstruction from rectified images Disparity: d=u’-u B: baseline between O and O’ Depth: z = -B/d × f

Correlation methods (1970--) Slide the window along the epipolar line until w.w’ is maximized. 2 Minimize ||w-w’|| Normalized Correlation: minimize q instead.

Forshortening problems Solution: add a second pass using disparity estimates to warp the correlation windows, e.g. Devernay and Faugeras (1994)

Multiscale edge matching [Marr, Poggio and Grimson, 1979-81] Edges are found by repeatedly smoothing the image and detecting the zero crossings of the second derivative (Laplacian) Matches at coarse scales are used to offset the search for matches at fine scales (equivalent to eye movements)

Multiscale edge matching One of the two input images Image Laplacian Zeros of the Laplacian

Multiscale edge matching

Ordering constraints In general the points are in the same order on both epipolar lines. But it is not always the case..

Ordering constraints points are not necessarily in order d-b-a c’-b’-d’

Dynamic programming Find the minimum-cost path going monotonically [Baker and Binford, 1981] Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

Dynamic programming [Ohta and Kanade, 1985] use inter-scanline f or point correspondence on vertical edges [Ohta and Kanade, 1985] use DP for both intra-scanline and inter-scanline search

Three views The third eye can be used for verification.. b1-a2 match is wrong as thee is no corresponding point in camera 3

More views Pick a reference image, and slide the corresponding window along the corresponding epipolar lines of all other images, using inverse depth relative to the first image as the search parameter. [Okutami and Kanade, 1993] Use the sum of correlation scores to rank matches.

I1 I2 I10

Correspondence Extensive literature Smooth surfaces Region based Graph cut Mutual information Belief propagation Conditional random field Smooth surfaces

Middlebury stereo dataset De facto data set with ground truth, code, and comprehensive performance evaluation http://vision.middlebury.edu/stereo/data/

Performance evaluation