Computer Vision, Robert Pless our goal is to understand the process of stereo vision. Last time, we studied the “Essential” Matrix, which describe where a point in one image may appear in a second image. Computer Vision, Robert Pless

Computer Vision, Robert Pless Rectification Need to rotate each camera so the epipole is along the y-axis. (0,+/-1,0) Is this unique? Computer Vision, Robert Pless

Stereo Image Rectification Computer Vision, Robert Pless

Computer Vision, Robert Pless

Stereo Image Rectification reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999. Computer Vision, Robert Pless

Computer Vision, Robert Pless Stereo Rectification Computer Vision, Robert Pless

Computer Vision, Robert Pless Rectification Computer Vision, Robert Pless

Basic Stereo Algorithm Improvement: match windows This should look familar... Correlation, Sum of Squared Difference (SSD), etc. For each pixel in the left image For each epipolar line compare with every pixel on same epipolar line in right image pick pixel with minimum match cost Computer Vision, Robert Pless

Computer Vision, Robert Pless Then, finally, P Z Disparity: xl xr f pl pr Ol Or b Then given Z, we can compute X and Y. b is the stereo baseline d measures the difference in retinal position between corresponding points Computer Vision, Robert Pless

Using these constraints we can use matching for stereo Improvement: match windows For each pixel in the left image For each epipolar line compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This will never work, so: Computer Vision, Robert Pless

Correspondence: Photometric constraint Same world point has same intensity in both images. Lambertian fronto-parallel Issues: Noise Specularity Foreshortening Computer Vision, Robert Pless

Computer Vision, Robert Pless = ? f g Comparing Windows: Most popular For each window, match to closest window on epipolar line in other image. Computer Vision, Robert Pless

Computer Vision, Robert Pless Window size W = 3 W = 20 Effect of window size Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 smaller window: more detail, more noise bigger window: less noise, more detail Computer Vision, Robert Pless

Computer Vision, Robert Pless Stereo results Data from University of Tsukuba Scene Ground truth Computer Vision, Robert Pless

Results with window correlation Window-based matching (best window size) Ground truth Computer Vision, Robert Pless

Results with better method State of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Ground truth Computer Vision, Robert Pless

Computer Vision, Robert Pless Stereo Results Trinocular stereo system available from Point Gray Research for $5K (circa ’97) Computer Vision, Robert Pless

Computer Vision, Robert Pless Stereo Example [Szeliski & Kang ‘95] depth map 3D rendering input image (1 of 2) Computer Vision, Robert Pless

Computer Vision, Robert Pless Stereo Example left image right image depth map H. Tao et al. “Global matching criterion and color segmentation based stereo” Computer Vision, Robert Pless

Computer Vision, Robert Pless Stereo Example Computer Vision, Robert Pless H. Tao et al. “Global matching criterion and color segmentation based stereo”

What are these additional constraints? corresponding points along corresponding epipolar lines Computer Vision, Robert Pless

Order ! Once the epipolar geometry is known, we can increase constraints on potential correspondences further... corresponding points along corresponding epipolar lines 3d points giving rise to those images Computer Vision, Robert Pless

Order ! Out-of-order points indicate “seeing around” an object, and generally doesn’t occur ... corresponding points along corresponding epipolar lines 3d points giving rise to those images Can you see the dynamic programming problem here? Computer Vision, Robert Pless

Computer Vision, Robert Pless Correspondence It is fundamentally ambiguous, even with stereo constraints Ordering constraint… …and its failure Computer Vision, Robert Pless

Search Over Correspondences Occluded Pixels Left scanline Right scanline Dis-occluded Pixels Three cases: Sequential – cost of match Occluded – cost of no match Disoccluded – cost of no match Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Start Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Dis-occluded Pixels Right scanline End Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Dis-occluded Pixels Right scanline Terminal Computer Vision, Robert Pless

Computing Correspondence Another approach is to match edges rather than windows of pixels: Which method is better? Edges tend to fail in dense texture (outdoors) Correlation tends to fail in smooth featureless areas Computer Vision, Robert Pless

Computing Correspondences Both methods fail for smooth surfaces There is currently no good solution to the correspondence problem Computer Vision, Robert Pless

History and Philosophy of human stereo Features vs. Pixels? Do we extract features prior to matching? Julesz-style Random Dot Stereogram Computer Vision, Robert Pless

Computer Vision, Robert Pless Human abilities Computer Vision, Robert Pless

Computer Vision, Robert Pless Human abilities Computer Vision, Robert Pless

Computer Vision, Robert Pless Assignment: Stereo In or Stereo Out (2 options) Write a program that takes as input a rectified stereo pair, and creates the 3D model of the scene. Show your program output on: http://svs.gsfc.nasa.gov/vis/a000000/a003500/a003531/ Write a program that takes as input a depth map, and creates an auto-stereogram that gives that depth map. Google search for “depth map” Computer Vision, Robert Pless