Presentation on theme: "Stereo Vision Reading: Chapter 11"— Presentation transcript:
1 Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more imagesSubproblems:Calibrating camera positions.Finding all corresponding points (hardest part)Computing depth or surfaces.Slide credits for this chapter: David Jacobs, Frank Dellaert, Octavia Camps, Steve Seitz
3 Stereo visionTriangulate on two images of the same point to recover depth.Feature matching across viewsCalibrated camerasdepthbaselineLeftRightMatching correlationwindows across scan lines
4 The epipolar constraint epipolar planeepipolar lineepipolar lineEpipolar ConstraintMatching points lie along corresponding epipolar linesReduces correspondence problem to 1D search along conjugate epipolar linesGreatly reduces cost and ambiguity of matchingSlide credit: Steve Seitz
5 Simplest Case: Rectified Images Image planes of cameras are parallel.Focal points are at same height.Focal lengths same.Then, epipolar lines fall along the horizontal scan lines of the imagesWe will assume images have been rectified so that epipolar lines correspond to scan linesSimplifies algorithmsImproves efficiency
6 We can always achieve this geometry with image rectification Image Reprojectionreproject image planes onto common plane parallel to line between optical centersNotice, only focal point of camera really matters(Seitz)
8 CorrespondenceIt is fundamentally ambiguous, even with stereo constraintsOrdering constraint……and its failure
9 Correspondence: What should we match? Objects?Edges?Pixels?Collections of pixels?
10 Julesz: showed that recognition is not needed for stereo.
11 Correspondence: Epipolar constraint. The epipolar constraint helps, but much ambiguity remains.
12 Correspondence: Photometric constraint Same world point has same intensity in both images.True for Lambertian surfacesA Lambertian surface has a brightness that is independent of viewing angleViolations:NoiseSpecularityNon-Lambertian materialsPixels that contain multiple surfaces
13 Pixel matching For each epipolar line For each pixel in the left image Improvement: match windowsFor each epipolar linecompare with every pixel on same epipolar line in right imagepick pixel with minimum match costThis leaves too much ambiguity, so:(Seitz)
14 Correspondence Using Correlation LeftRightscanlineSSD errordisparityLeftRight
16 Image NormalizationEven when the cameras are identical models, there can be differences in gain and sensitivity.For these reason and more, it is a good idea to normalize the pixels in each window:
17 Images as VectorsLeftRight“Unwrap” image to form vector, using raster scan orderrow 1row 2Each window is a vector in an m2 dimensional vector space. Normalization makes them unit length.row 3
18 Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation
19 Stereo ResultsImages courtesy of Point Grey Research
20 Window size W = 3 W = 20 Effect of window size Some approaches have been developed to use an adaptive window size (try multiple sizes and select best match)smaller window: more detail, more noisebigger window: less noise, more detail(Seitz)
21 Stereo testing and comparisons D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), ppSceneGround truth
23 Results with window correlation Window-based matching(best window size)Ground truth(Seitz)
24 Results with better method State of the art method: Graph cutsGround truth(Seitz)
25 Stereo Correspondences Left scanlineRight scanline…
26 Stereo Correspondences Left scanlineRight scanline…OcclusionMatchMatchMatchDisocclusion
27 Search Over Correspondences Occluded PixelsLeft scanlineRight scanlineDisoccluded PixelsThree cases:Sequential – add cost of match (small if intensities agree)Occluded – add cost of no match (large cost)Disoccluded – add cost of no match (large cost)
28 Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineStartDynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraintDis-occluded PixelsRight scanlineEnd
29 Dynamic ProgrammingEfficient algorithm for solving sequential decision (optimal path) problems.1111…22223333How many paths through this trellis?
30 Dynamic Programming 1 1 1 States: 2 2 2 3 3 3 Suppose cost can be decomposed into stages:
31 Dynamic Programming111222333Principle of Optimality for an n-stage assignment problem:
33 Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineScan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.Dis-occluded PixelsRight scanlineTerminal
34 Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineScan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.Dis-occluded PixelsRight scanlineTerminal
35 Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineScan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.Dis-occluded PixelsRight scanlineTerminal
39 Result using a good technique This is a stereo pair captured in our lab.This is the disparity obtained by our approach.Notice that the thin structures and narrow holes are accounted for.DisparityLeft ImageRight Image
40 View InterpolationThis is the result of view interpolating using extracted depth information.
41 Computing Correspondence Another approach is to match edges rather than windows of pixels:Which method is better?Edges tend to fail in dense texture (outdoors)Correlation tends to fail in smooth featureless areas
42 Summary of different stereo methods Constraints:Geometry, epipolar constraint.Photometric: Brightness constancy, only partly true.Ordering: only partly true.Smoothness of objects: only partly true.Algorithms:What you compare: points, regions, features?How you optimize:Local greedy matches.1D search.2D search.