Bahadir K. Gunturk1 Phase Correlation

Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform  Location of the impulse function gives the translation amount between the images

Bahadir K. Gunturk3 Phase Correlation

Computer Vision Stereo Vision

Bahadir K. Gunturk5 Coordinate Systems Let O be the origin of a 3D coordinate system spanned by the unit vectors i, j, and k orthogonal to each other. i j k O P Coordinate vector

Bahadir K. Gunturk6 Homogeneous Coordinates n H O P Homogeneous coordinates

Bahadir K. Gunturk7 Coordinate System Changes Translation

Bahadir K. Gunturk8 Coordinate System Changes Rotation where Exercise: Write the rotation matrix for a 2D coordinate system.

Bahadir K. Gunturk9 Coordinate System Changes Rotation + Translation

Bahadir K. Gunturk10 Perspective Projection Perspective projection equations

Bahadir K. Gunturk11 Review: Pinhole Camera

Bahadir K. Gunturk12 Review: Perspective Projection

Bahadir K. Gunturk13 Multi-View Geometry Relates 3D World Points Camera Centers Camera Orientations Camera Parameters Image Points

Bahadir K. Gunturk14 Stereo scene point optical center image plane p p’

Bahadir K. Gunturk15 Finding Correspondences p p’

Bahadir K. Gunturk16 Three Questions Correspondence geometry: Given an image point p in the first view, how does this constrain the position of the corresponding point p’ in the second? Camera geometry (motion): Given a set of corresponding image points {p i ↔ p’ i }, i=1,…,n, what are the cameras C and C’ for the two views? Or what is the geometric transformation between the views? Scene geometry (structure): Given corresponding image points p i ↔ p’ i and cameras C, C’, what is the position of the point X in space?

Bahadir K. Gunturk17 Stereo Constraints X1X1 Y1Y1 Z1Z1 O1O1 Image plane Focal plane M p p’ Y2Y2 X2X2 Z2Z2 O2O2 Epipolar Line Epipole

Bahadir K. Gunturk18 Epipolar Constraint

Bahadir K. Gunturk19 From Geometry to Algebra O O’ P p p’ All vectors shown lie on the same plane.

Bahadir K. Gunturk20 From Geometry to Algebra O O’ P p p’

Bahadir K. Gunturk21 Matrix form of cross product a×b=|a||b|sin(η)u a=a x i+a y j+a z k b=b x i+b y j+b z k

Bahadir K. Gunturk22 The Essential Matrix Essential matrix

Bahadir K. Gunturk23 Stereo Vision Two cameras. Known camera positions. Recover depth.

Bahadir K. Gunturk24 Recovering Depth Information O2O2O2O2 P’ 2 =Q’ 2 P Q O1O1O1O1 P’ 1 Q’ 1 Depth can be recovered with two images and triangulation.

Bahadir K. Gunturk25 A Simple Stereo System Z w =0 LEFT CAMERA Left image: reference Right image: target RIGHT CAMERA Elevation Z w disparity Depth Z baseline

Bahadir K. Gunturk26 Stereo View Left View Right View Disparity

Bahadir K. Gunturk27 Stereo Disparity The separation between two matching objects is called the stereo disparity.

Bahadir K. Gunturk28 Parallel Cameras OlOlOlOl OrOrOrOr P plplplpl prprprpr T Z xlxlxlxl xrxrxrxr f T is the stereo baseline Disparity:

Bahadir K. Gunturk29 Disparity Equation P(X,Y,Z) p l (x l,y l ) Optical Center O l f = focal length Image plane LEFT CAMERA T = Baseline Depth Stereo system with parallel optical axes f = focal length Optical Center O r p r (x r,y r ) Image plane RIGHT CAMERA

Bahadir K. Gunturk30 Disparity vs. Baseline P(X,Y,Z) p l (x l,y l ) Optical Center O l f = focal length Image plane LEFT CAMERA T = Baseline Depth f = focal length Optical Center O r p r (x r,y r ) Image plane RIGHT CAMERA Disparity Stereo system with parallel optical axes

Bahadir K. Gunturk31 Finding Correspondences

Bahadir K. Gunturk32 Correlation Approach For Each point (x l, y l ) in the left image, define a window centered at the point (x l, y l ) LEFT IMAGE

Bahadir K. Gunturk33 Correlation Approach … search its corresponding point within a search region in the right image (x l, y l ) RIGHT IMAGE

Bahadir K. Gunturk34 Correlation Approach … the disparity (dx, dy) is the displacement when the correlation is maximum (x l, y l )dx(x r, y r ) RIGHT IMAGE

Bahadir K. Gunturk35 Stereo correspondence Epipolar Constraint  Reduces correspondence problem to 1D search along epipolar lines epipolar plane epipolar line

Bahadir K. Gunturk36 For each epipolar line For each pixel in the left image Compare with every pixel on same epipolar line in right image Pick pixel with the minimum matching error Of course, matching single pixels won’t work; so, we match regions around pixels. Stereo correspondence

Bahadir K. Gunturk37 Comparing Windows =?f g Mostpopular For each window, match to closest window on epipolar line in other image.

Bahadir K. Gunturk38 Maximize Cross correlation Minimize Sum of Squared Differences Comparing Windows

Bahadir K. Gunturk39 Feature-based correspondence Features most commonly used:  Corners Similarity measured in terms of:  surrounding gray values (SSD, Cross-correlation)  location  Edges, Lines Similarity measured in terms of:  orientation  contrast  coordinates of edge or line’s midpoint  length of line

Bahadir K. Gunturk40 Feature-based Approach For each feature in the left image… LEFT IMAGE corner line structure

Bahadir K. Gunturk41 Feature-based Approach Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum RIGHT IMAGE corner line structure

Bahadir K. Gunturk42 Correspondence Difficulties Why is the correspondence problem difficult?  Some points in each image will have no corresponding points in the other image. (1) the cameras might have different fields of view. (2) due to occlusion. A stereo system must be able to determine the image parts that should not be matched.

Bahadir K. Gunturk43 Structured Light Structured lighting  Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult).  Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth.  Finding and matching such points is simplified by knowing the geometry of the projected patterns.

Bahadir K. Gunturk44 Stereo results Ground truthScene  Data from University of Tsukuba (Seitz)

Bahadir K. Gunturk45 Results with window correlation Estimated depth of field (a fixed-size window) Ground truth (Seitz)

Bahadir K. Gunturk46 Results with better method A state of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September Ground truth (Seitz)

Bahadir K. Gunturk47 Window size W = 3W = 20 Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998Stereo matching with nonlinear diffusion Effect of window size (Seitz)

Bahadir K. Gunturk48 Other constraints It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t change too quickly.)

Bahadir K. Gunturk49 Parameters of a Stereo System Intrinsic Parameters  Characterize the transformation from camera to pixel coordinate systems of each camera  Focal length, image center, aspect ratio Extrinsic parameters  Describe the relative position and orientation of the two cameras  Rotation matrix R and translation vector T p l p r P OlOl OrOr XlXl XrXr PlPl PrPr flfl frfr ZlZl YlYl ZrZr YrYr R, T

Bahadir K. Gunturk50 Applications courtesy of Sportvision First-down line

Bahadir K. Gunturk51 Applications Virtual advertising courtesy of Princeton Video Image