Presentation on theme: "Stereo Vision Reading: Chapter 11"— Presentation transcript:
1Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more imagesSubproblems:Calibrating camera positions.Finding all corresponding points (hardest part)Computing depth or surfaces.Slide credits for this chapter: David Jacobs, Frank Dellaert, Octavia Camps, Steve Seitz
3Stereo visionTriangulate on two images of the same point to recover depth.Feature matching across viewsCalibrated camerasdepthbaselineLeftRightMatching correlationwindows across scan lines
4The epipolar constraint epipolar planeepipolar lineepipolar lineEpipolar ConstraintMatching points lie along corresponding epipolar linesReduces correspondence problem to 1D search along conjugate epipolar linesGreatly reduces cost and ambiguity of matchingSlide credit: Steve Seitz
5Simplest Case: Rectified Images Image planes of cameras are parallel.Focal points are at same height.Focal lengths same.Then, epipolar lines fall along the horizontal scan lines of the imagesWe will assume images have been rectified so that epipolar lines correspond to scan linesSimplifies algorithmsImproves efficiency
6We can always achieve this geometry with image rectification Image Reprojectionreproject image planes onto common plane parallel to line between optical centersNotice, only focal point of camera really matters(Seitz)
8CorrespondenceIt is fundamentally ambiguous, even with stereo constraintsOrdering constraint……and its failure
9Correspondence: What should we match? Objects?Edges?Pixels?Collections of pixels?
10Julesz: showed that recognition is not needed for stereo.
11Correspondence: Epipolar constraint. The epipolar constraint helps, but much ambiguity remains.
12Correspondence: Photometric constraint Same world point has same intensity in both images.True for Lambertian surfacesA Lambertian surface has a brightness that is independent of viewing angleViolations:NoiseSpecularityNon-Lambertian materialsPixels that contain multiple surfaces
13Pixel matching For each epipolar line For each pixel in the left image Improvement: match windowsFor each epipolar linecompare with every pixel on same epipolar line in right imagepick pixel with minimum match costThis leaves too much ambiguity, so:(Seitz)
14Correspondence Using Correlation LeftRightscanlineSSD errordisparityLeftRight
16Image NormalizationEven when the cameras are identical models, there can be differences in gain and sensitivity.For these reason and more, it is a good idea to normalize the pixels in each window:
17Images as VectorsLeftRight“Unwrap” image to form vector, using raster scan orderrow 1row 2Each window is a vector in an m2 dimensional vector space. Normalization makes them unit length.row 3
18Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation
19Stereo ResultsImages courtesy of Point Grey Research
20Window size W = 3 W = 20 Effect of window size Some approaches have been developed to use an adaptive window size (try multiple sizes and select best match)smaller window: more detail, more noisebigger window: less noise, more detail(Seitz)
21Stereo testing and comparisons D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), ppSceneGround truth
23Results with window correlation Window-based matching(best window size)Ground truth(Seitz)
24Results with better method State of the art method: Graph cutsGround truth(Seitz)
25Stereo Correspondences Left scanlineRight scanline…
26Stereo Correspondences Left scanlineRight scanline…OcclusionMatchMatchMatchDisocclusion
27Search Over Correspondences Occluded PixelsLeft scanlineRight scanlineDisoccluded PixelsThree cases:Sequential – add cost of match (small if intensities agree)Occluded – add cost of no match (large cost)Disoccluded – add cost of no match (large cost)
28Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineStartDynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraintDis-occluded PixelsRight scanlineEnd
29Dynamic ProgrammingEfficient algorithm for solving sequential decision (optimal path) problems.1111…22223333How many paths through this trellis?
30Dynamic Programming 1 1 1 States: 2 2 2 3 3 3 Suppose cost can be decomposed into stages:
31Dynamic Programming111222333Principle of Optimality for an n-stage assignment problem:
33Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineScan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.Dis-occluded PixelsRight scanlineTerminal
34Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineScan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.Dis-occluded PixelsRight scanlineTerminal
35Stereo Matching with Dynamic Programming Occluded PixelsLeft scanlineScan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.Dis-occluded PixelsRight scanlineTerminal
39Result using a good technique This is a stereo pair captured in our lab.This is the disparity obtained by our approach.Notice that the thin structures and narrow holes are accounted for.DisparityLeft ImageRight Image
40View InterpolationThis is the result of view interpolating using extracted depth information.
41Computing Correspondence Another approach is to match edges rather than windows of pixels:Which method is better?Edges tend to fail in dense texture (outdoors)Correlation tends to fail in smooth featureless areas
42Summary of different stereo methods Constraints:Geometry, epipolar constraint.Photometric: Brightness constancy, only partly true.Ordering: only partly true.Smoothness of objects: only partly true.Algorithms:What you compare: points, regions, features?How you optimize:Local greedy matches.1D search.2D search.