Multiview geometry ECE 847: Digital Image Processing Stan Birchfield Clemson University

Stereo geometry for pinhole cameras with flat retinas C,C’,x,x’ and X are coplanar Left cameraRight camera world point center of projection epipolar plane epipolar line for x epipole baseline M. Pollefeys,

epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) Epipolar geometry M. Pollefeys,

What if only C,C’,x are known? Epipolar geometry epipole center of projection baseline All points on p project on l and l’ M. Pollefeys,

Family of planes  and lines l and l’ Intersection in e and e’ Epipolar geometry M. Pollefeys,

Example: Converging cameras M. Pollefeys,

e e’ Example: Forward motion M. Pollefeys,

Epipolar geometry and Fundamental matrix epipolar line (epipole: intersection of all epipolar lines) (computed with 20 points, non-normalized algorithm)

(computed with 50 points, non-normalized algorithm) Epipolar geometry and Fundamental matrix epipolar line (epipole: intersection of all epipolar lines)

Epipolar geometry and Fundamental matrix using OpenCV implementation best

Fundamental matrix point in image 1 point in image 2 fundamental matrix

Epipolar lines (1) epipolar line in image 2 associated with (x,y) in image 1

Epipolar lines (2) epipolar line in image 1 associated with (x’,y’) in image 2

Computing the fundamental matrix (Eight-point algorithm) known unknown

known unknown Computing the fundamental matrix (Eight-point algorithm)

1.Construct A (nx9) from correspondences 2.Compute SVD of A: A = U  V T 3.Unit-norm solution of Af=0 is given by v n (the right-most singular vector of A) 4.Reshape f into F 1 5.Compute SVD of F 1 : F 1 =U F  F V F T 6.Set  F (3,3)=0 to get  F ’ (The enforces rank(F)=2) 7.Reconstruct F=U F  F ’V F T 8.Now x T F is epipolar line in image 2, and Fx’ is epipolar line in image 1 Note: Need to normalize coordinates to get good results Computing the fundamental matrix (Eight-point algorithm)

Normalizing coordinates Be careful: order of x and x’ has switched!

Normalizing coordinates Step 1: Compute normalizing transformation matrices: Step 2: Normalize coordinates: Step 3: Use 8-point algorithm to compute : Step 4: Transform back to original coordinate system:

(simple for stereo  rectification) Example: Motion parallel to image plane M. Pollefeys,

Example: Motion parallel to image scanlines Epipoles are at infinity Scanlines are the epipolar lines In this case, the images are said to be “rectified” Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba

Standard (rectified) stereo geometry pure translation along X-axis M. Pollefeys,

Perspective projection X x X x M. Pollefeys,

Rectified geometry xLxL xRxR XLXL -X R b

Standard stereo geometry disparity is inversely proportional to depth stereo vision is less useful for distant objects M. Pollefeys,

Stereo camera configurations (Slide from Pascal Fua)

More cameras Multi-baseline stereo [Okutomi & Kanade]

Applications Take 3 pictures, reconstruct 3D geometry

3D coordinates of cardboard cutouts (u,v)  (x,y,z) (u,v 0 )  (x,y,0)(u 1,v 0 )  (x 1,y 1,0) z =√ (x 1 -x) 2 +(y 1 -y) 2 Assuming vertical object, vertical image plane, and unity aspect ratio: