1. CS 223b 1 More on stereo and correspondence

2. CS 223b 2 =?f g Mostpopular For each window, match to closest window on epipolar line in other image. (slides O. Camps) Comparing Windows:

3. CS 223b 3 Window size W = 3W = 20 Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998Stereo matching with nonlinear diffusion Effect of window size (S. Seitz)

4. CS 223b 4 Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

5. CS 223b 5 Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

6. CS 223b 6 Dynamic Programming (Ohta and Kanade, 1985) Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2):139-154 (1985).  1985 IEEE.

7. CS 223b 7 Other constraints Smoothness: disparity usually doesn’t change too quickly. Unfortunately, this makes the problem 2D again. Solved with a host of graph algorithms, Markov Random Fields, Belief Propagation, …. Uniqueness constraint (each feature can at most have one match) Occlusion and disparity are connected.

8. CS 223b 8 Feature-based Methods Conceptually very similar to Correlation-based methods, but: They only search for correspondences of a sparse set of image features. Correspondences are given by the most similar feature pairs. Similarity measure must be adapted to the type of feature used.

9. CS 223b 9 Feature-based Methods: Features most commonly used: Corners Similarity measured in terms of: surrounding gray values (SSD, Cross-correlation) location Edges, Lines Similarity measured in terms of: orientation contrast coordinates of edge or line’s midpoint length of line

10. CS 223b 10 Example: Comparing lines l l and l r : line lengths  l and  r : line orientations (x l,y l ) and (x r,y r ): midpoints c l and c r : average contrast along lines  l    m  c : weights controlling influence The more similar the lines, the larger S is!

11. CS 223b 11 Correspondence By Features RIGHT IMAGE corner line structure Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

12. CS 223b 12 Dense Stereo Matching: Examples View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade’88]

13. CS 223b 13 Dense Stereo Matching Some other view extrapolation results inputdepth imagenovel view

14. CS 223b 14 Dense Stereo Matching Compute certainty map from correlations input depth map certainty map

15. CS 223b 15 Ordering constraint Usually, order of points in two images is same. Is this always true? If we match pixel i in image 1 to pixel j in image 2, no matches that follow will affect which are the best preceding matches. Example with pixels (a la Cox et al.).

16. CS 223b 16 The Ordering Constraint But it is not always the case... This enables dynamic programming Points on the epipolar lines appear in the same order

17. CS 223b 17 Correspondence Problem 2 Correspondence fail for smooth surfaces There is currently no good solution to the correspondence problem

18. CS 223b 18 Correspondence Problem 3 Regions without texture Highly Specular surfaces Translucent objects

19. CS 223b 19 Stereo Vision: Outline Basic Equations Epipolar Geometry Image Rectification Reconstruction Correspondence Active Range Imaging Technology Dense and Layered Stereo Smoothing With Markov Random Fields

20. CS 223b 20 How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy)

21. CS 223b 21 Prof Marc Levoy @ Stanford By James Davis, Honda Research, Now UCSC

22. CS 223b 22 rectified Active Stereo (Structured Light)

23. CS 223b 23 DP for Correspondence Does this always work? When would it fail? Failure Example 1 Failure Example 2 Failure Example 3

24. CS 223b 24 Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

25. CS 223b 25 Stereo Correspondences …… Left scanlineRight scanline

26. CS 223b 26 Search Over Correspondences Three cases: Sequential – cost of match Occluded – cost of no match Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

27. CS 223b 27 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

28. CS 223b 28 Stereo Matching with Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

29. CS 223b 29 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

30. CS 223b 30 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming