1. John Morris Iolanthe returns to the Waitemata Harbour Stereo Vision The Correspondence Problem

2. 2 Correspondence Problem Methods for establishing correspondences −Two issues −How to select candidate matches? −How to determine the goodness of a match? −Two main classes of correspondence (matching) algorithm: −Correlation-based −Attempt to establish a correspondence by matching image intensities – usually over a window of pixels in each image  Dense disparity maps −Distance is found for all BV image points −Except occluded (MV) points −Feature-based −Attempt to establish a correspondence by matching a sparse sets of image features – usually edges −Disparity map is sparse −Number of points is related to the number of image features identified

3. 3 Feature-Based Methods Main idea −Look for a feature in an image that matches a feature in the other. −Typical features used are: −edge points −line segments −corners (junctions)

4. 4 Feature-Based Methods A set of features is used for matching −a line feature descriptor, for example, could contain: −length, l −orientation,  −coordinates of the midpoint, m −average intensity along the line, i Similarity measures are based on matching feature descriptors: where w 0,..., w 3 are weights (determining the weights that yield the best matches is a nontrivial task).

5. 5 Feature-Based Methods

6. 6 Correlation vs. feature-based approaches Correlation methods −Easier to implement −Provide a dense disparity map (useful for reconstructing surfaces) −Need textured images to work well (many false matches otherwise) −Don’t work well when viewpoints are very different, due to −change in illumination direction −violates Lambertian scattering assumption −foreshortening −perspective problem – surfaces are not fronto-planar Feature-based methods −Suitable when good features can be extracted from the scene −Faster than correlation-based methods −Provide sparse disparity maps −OK for applications like visual navigation −Relatively insensitive to illumination changes

7. 7 Feature-based Methods Matching is difficult! Selecting features should make it easier? but Correlation techniques have received more attention Possible reasons −Feature detection is sensitive to noise −Features ‘appear’ to have quite different geometries −Lengths, local curvatures differ −Edges easily split into two or more edges −Feature area altered by perspective −Failure of the fronto-planar assumption −Texture on flat surfaces generates unnecessary features −…

8. 8 Correlation Matching Fundamental algorithm −Canonical configuration (  image planes,  optical axes) −Search along scan lines  d = x L - x R Problem −How do we find the ‘best’ match? For each pixel in one image (the reference image) Find best match (best correlation) in the other image Use the disparity, d, to determine depth, Z Z  1/d

9. 9 Real images are noisy! A.Signal noise arising from electromagnetic interference ( eg cross-talk), quantum behaviour of electronic devices ( eg resistor shot-noise) and quantization noise from digitizing real-valued analogue signals B.Geometric sources Discrete pixel sensors with finite area, Occlusions, Perspective distortion C.Electronic sources intensity sensitivity variations between cameras ( eg different optical or electronic gain settings), different `dark noise' levels D.Optical sources non-uniform scattering (non-Lambertian surfaces), reflections and specular highlights, angle dependent colour scattering (`grating' effects), lighting variation due to different view angles

10. 10 Noise in Stereo Matching A.Signal noise Common to almost all electronic measurement equipment and introduce random perturbations to measured image intensities B.Geometric sources Arise from the internal structure of digital cameras themselves and the stereo system configuration. C.Electronic sources Some configurations avoid this noise by using a single camera on a translation base or a moving scene ( eg object on a rotation stage). D.Optical sources The physical separation of the two cameras results in different viewing angles for the scene and produces this group of problems, caused by assumptions usually made by matching algorithms - all surfaces are perfect Lambertian scatterers.

11. 11 Solutions – Average the Noise Window-based Correlation Algorithms Compare a window of pixels in one image with a window of pixels in the other −Noise averages itself out over the window Reference Image d=0 d=d= Move an identical window along the epipolar line In the other image looking for the best match Disparity, d, varies from 0 ( Z =  ) to  ( Z = min ) where min is determined by system configuration – closest approach of an object

12. 12 Correlation-Based Methods

13. 13 Correlation-Based Methods Basic Algorithm −Assume rectified images in canonical configuration −Epipolar lines aligned with scan lines or −Conjugate pairs lie in corresponding scan lines for each row, k for j =  to w c min =  for d = 0 to  // check each possible disparity c(d) = f ( I 1 (j,k), I 2 (j-d,k) ) if c(d) < c min then d best = d c min = c(d) disp( j,k ) = d best // Save best d value

14. 14 Correlation-Based Methods Cost function −Simplest f ( I 1 (j,k), I 2 (j-d,k) ) = | I 1 (j,k) - I 2 (j-d,k) | absolute difference −Generally a window of pixels around j,k will be considered f ( I 1 (j,k), I 2 (j-d,k) ) =  | I 1 (j+p,k+q) - I 2 (j+p-d,k+q) | − SAD takes random pixel noise into account well − Gain (contrast) and offset deviations may be partially taken into account by normalizing over the window  -w≤p≤+w -w≤q≤+w Sum of Absolute Differences (SAD)

15. 15 Correlation-Based Methods Usually, we normalize c(d) by dividing it by the standard deviation of both I l and I r (normalized cross-correlation, c(d)  [0,1]) where and are the average pixel values in the left and right windows. An alternative similarity measure is the sum of squared differences (SSD): But experiment shows that the simpler sum of absolute differences (SAD) is just as good for reasonable c(d) =   | I l (i+k,j+l) – I r (i+k-d,j+l) |

16. 16 Correlation-Based Methods Experiment shows that the simpler sum of absolute differences (SAD) is just as good for reasonable cameras (Gains and offsets matched) Stereo matching is computationally expensive! Correlation algorithms have time complexity O( h w  (win) 2 ) Even though the pixel comparison is simple and fast (absolute difference), for a medium resolution image (h = w = 10 3 ), reasonable depth accuracy (1%,  > 100) and sufficiently large windows for good matching accuracy, win  10, h w  (win) 2 = 10 3  10 3  10 2  10  10 = 10 10 The total number of operations required for a single image is quite large! More complex cost functions – SSD, normalized correlation, etc – which involve more complex computation are often not justified!

17. 17 Correlation-Based Methods Comments −The success of correlation-based methods depends on whether the image window in one image exhibits a distinctive structure that occurs infrequently in the search region of the other image. −How to choose the size of the window, W? −too small a window −may not capture enough image structure and −may be too noise sensitive  many false matches −too large a window −makes matching less sensitive to noise (desired) but also −decreases precision (blurs disparity map) −An adaptive searching window has been proposed: T Kanade and M Okutomi, A stereo matching algorithm with an adaptive window, IEEE Trans PAMI, 16(9), 920-932 (1994)

18. 18 Correlation Methods – Adaptive Windows Input – Ground truth 3x3 window Too noisy! 7x7 window Sharp edges are blurred! Adaptive window Sharp edges and less noise

19. 19 Correlation-Based Methods Improvements −Instead of using the image intensity values, the accuracy of correlation is improved by using thresholded signed gradient magnitudes at each pixel. −Compute the gradient magnitude at each pixel in the two images without smoothing −Map the gradient magnitude values into three values: -1, 0, 1 (by thresholding the gradient magnitude) −More sensitive correlations are produced this way + several dozen more see + D Scharstein & R Szeliski, Intl Journal of Computer Vision, 47(1), 7-42 (2001) for a review +This is quite an important paper: it classifies many of the approaches to stereo matching and provides an objective quantitative comparison of many algorithms! +The use of metrics to compare approaches appears to be quite rare in computer vision papers – another reason to read this one!!

20. 20 Correlation-Based Methods + several dozen more see + D Scharstein & R Szeliski, Intl Journal of Computer Vision, 47(1), 7-42 (2001) for a review +This is quite an important paper: it classifies many of the approaches to stereo matching and provides an objective quantitative comparison of many algorithms! +The use of metrics to compare approaches appears to be quite rare in computer vision papers – another reason to read this one!! +Middlebury stereo pages +Daniel Scharstein, Middlebury College, Vermont + http://vision.middlebury.edu/stereo/ http://vision.middlebury.edu/stereo/ +Comparisons of many stereo matching algorithms

21. 21 Other correspondence algorithms Dynamic programming (Gimel’farb) −Finds a ‘path’ through an image which provides the best (least-cost) match −Can allow for occlusions (Birchfield and Tomasi) −Generally provide better results than area-based correlation −Faster than correlation Graph Cut (Zabih et al) −Seems to provide best results −Very slow, not suitable for real-time applications Concurrent Stereo Matching −Examine all possible matches in parallel (Delmas, Gimel’farb, Morris, work in progress ) −Uses a model of image noise instead of arbitrary weights in cost functions −Suitable for real-time parallel hardware implementation Some of these will be considered in detail later

22. 22 Ordering Constraint If an object a is left on an object b in the left image then object a will also appear to the left of object b in the right image Ordering constraint……and its failure

23. 23 Correspondences …… Left scanlineRight scanline Match intensities sequentially between two scanlines

24. 24 Correspondences …… Left scanlineRight scanline Match Left occlusionRight occlusion

25. 25 Search Over Correspondences Three cases: −Sequential – cost of match −Left occluded – cost of no match −Right occluded – cost of no match Left scanline Right scanline Left Occluded Pixels Right occluded Pixels

26. 26 Standard 3-move Dynamic Programming for Stereo Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Left Occluded Pixels Left scanline Right occluded Pixels Right scanline Start End

27. 27 Dynamic Programming Efficient algorithm for solving sequential decision (optimal path) problems. 1 2 3 1 2 3 1 2 3 1 2 3 … How many paths through this trellis? i = 1 i = 2 i = 3 t = 1t = 2t = 3 t = T 3T3T

28. 28 Dynamic Programming 1 2 3 1 2 3 1 2 3 Suppose cost can be decomposed into stages: States: i = 1 i = 2 i = 3

29. 29 Dynamic Programming 1 2 3 1 2 3 1 2 3 Principle of Optimality for an n-stage assignment problem

30. 30 Dynamic Programming 1 2 3 1 2 3 1 2 3 Keep a record of the predecessor of each node

31. 31 Stereo Matching with Dynamic Programming Pseudo-code for calculating the optimal match M(i,j) : predecessors

32. 32 Stereo Matching with Dynamic Programming Pseudo-code for reconstructing the optimal path Occlusions – Skip until next match is found

33. 33 Dynamic Programming - Result Local errors may be propagated along a scan-line: no inter scan-line consistency is enforced

34. 34 Graph cut One of the best algorithms − S Roy, I J Cox, A maximum-flow formulation of N-camera stereo correspondence problem, Int Jnl Computer Vision, 34(2), 147- 161(1998) − Y Boykov, O Veksler, R Zabih, Fast approximate energy minimization via graph cuts, IEEE Trans PAMI, 23(11), 1222-1239 (2001) −Produces high correct match scores −Global −Computationally expensive

35. 35 Stereo As a Pixel-Labeling Problem Let P be a set of pixels, L be a label set. The goal is find a labeling f which minimize some energy. For stereo, the labels are disparities. The classic form of energy function is: Data term Smoothness term

36. 36 Energy Function: The energy function measures how appropriate a label is for the pixel p given the observed data. In stereo, this term corresponds to the match cost or likelihood. The energy term encodes the prior or smoothness constraint. In stereo, the so called Potts model is used:

37. 37 Two Energy Minimization Algorithm via Graph Cuts Swap algorithm

38. 38 Two Energy Minimization Algorithm via Graph Cuts expansion algorithm

39. 39 Moves

40. 40 Graph Cuts Results Graph Cuts Belief Propagation

41. 41 End of this section!

42. 42 Epipolar Geometry Significance of the epipolar lines −For an arbitrary stereo configuration, for each point (or window) in one image, you would need to search the whole of the other image for a match!  Very inefficient algorithm!  O(n 4 ) Left ImageRight Image ? n

43. 43 Epipolar Geometry ‘Canonical’ configuration −Optical axes, image planes & scan-lines parallel −Only necessary to search along scan lines −Corresponding point must lie on the same scan line in the other image  Simple (trivial) formulae for determining −Where to search −Same y coord −How to convert disparity to distance − Z  1 / d

44. 44 Epipolar Geometry General configuration −Optical axes verge on ‘fixation point’ in scene −Only necessary to search along epipolar lines −Corresponding point must lie on the corresponding epipolar line in the other image  More complex formulae −Where to search −Slope of epipolar line is a function of image coordinates −Distance from disparity − Z = f( x, y, d )

45. 45 Epipolar Geometry Neat demonstration! http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html Note the −Epipoles −Intersections of the baseline with the image planes −Fixed positions −At infinity in the canonical configuration −All the epipolar lines for one camera go through its epipole Epipolar Constraint −Corresponding points must lie on pairs of epipolar lines −Trucco refers to them as ‘conjugated epipolar lines’

46. 46 Assumptions and constraints Epipolar constraint −Corresponding points lie on corresponding epipolar lines −Holds for images if −Distortions are removed ie Cameras conform to pin-hole model −In the canonical configuration (|| optical axes, image planes) −Epipolar lines are scan lines  Simple software!  Rectification often used to transform images to canonical configuration −Images rotated and translated to a new view −Requires estimation of the fundamental matrix

47. 47 Assumptions and constraints Uniqueness constraint −Each pixel in the reference image corresponds to at most one pixel in the other image −Potential violations −Quantization of images into pixels −Corresponding ‘pixel’ actually spreads over several pixels −Reflections

48. 48 Assumptions and constraints Continuity constraint −Surfaces are generally continuous −Disparity differences between neighbouring pixels less than a threshhold −If x L 1 matches x R 1 and neighbouring pixel, x L 2 matches x R 2  || x L 1 – x R 1 | - | x L 2 – x R 2 || < th −Potential violations −Sharp edges −Only a small fraction of total image pixels −Can apply this constraint only in regions identified as belonging to one ‘object’ after segmentation

49. 49 Constraints Ordering constraint −Points on an epipolar line in one image appear on the corresponding epipolar line in the other image in the same order −Violations −Thin objects (‘poles’) well separated from a background Note the ordering of image points (lower case) in the left and right images

50. 50 Constraints Intensity −Intensities of matching points are the same −Gain and offset of both cameras identical −No noise Usually relaxed to −…… differ by a very small amount Fronto-planar −Surface segments −Subtend the same angle or −Occupy the same number of pixels in both images Violations Angled surfaces  Perspective problem