1. Stereo
Dan Kong

2. Stereo vision
Triangulate on two images of the same scene point to recover depth.
Camera calibration
Finding all correspondence
Computing depth or surfaces
depth
baseline
left
Right

3. Outline Basic stereo equations Constraints and assumption
Windows-based matching
Cooperative Stereo
Dynamic programming
Graph cut and Belief Propagation
Segmentation-based method

4. Pinhole Camera Model
Image
plane
Virtual
Image

5. Basic Stereo Derivations

6. Basic Stereo Derivations
disparity

7. Stereo Constraint Color constancy
The color of any world points remains constant from image to image
This assumption is true under Lambertian Model
In practice, given photometric camera calibration and typical scenes, color constancy holds well enough for most stereo algorithms.

8. Stereo Constraint Epipolar geometry
The epipolar geometry is the fundamental constraint in stereo.
Rectification aligns epipolar lines with scanlines
Epipolar plane
Epipolar line for p
Epipolar line for p’

9. Stereo Constraint Uniqueness and Continuity Proposed by Marr&Poggio.
Each item from each image may be assigned at most one disparity value,” and the “disparity” varies smoothly almost everywhere.

10. Correspondence Using Window-based matching
scanline
Left
Right
SSD error
disparity
Left
Right

11. Sum of Squared (Pixel) Differences
Left
Right

12. Image Normalization
Even when the cameras are identical models, there can be differences in gain and sensitivity.
The cameras do not see exactly the same surfaces, so their overall light levels can differ.
For these reasons and more, it is a good idea to normalize the pixels in each window:

13. Images as Vectors
Left
Right
“Unwrap” image to form vector, using raster scan order
row 1
row 2
Each window is a vector in an m2 dimensional vector space. Normalization makes them unit length.
row 3

14. Normalized Correlation

15. Results Using window-based Method
Left
Disparity Map
Images courtesy of Point Grey Research
Left
Right

16. Stereo Results
Left
Disparity map

17. Problems with Window-based matching
Disparity within the window must be constant.
Bias the results towards frontal-parallel surfaces.
Blur across depth discontinuities.
Perform poorly in textureless regions.
Erroneous results in occluded regions

18. Cooperative Stereo Algorithm
Based on two basic assumption by Marr and Poggio:
Uniqueness: at most a single unique match exists for each pixel.
Continuous: disparity values are generally continuous, i.e., smooth within a local neighborhood.

19. Disparity Space Image (DSI)
The 3D disparity space has dimensions row r column c and disparity d. Each element (r, c, d) of the disparity space projects to the pixel (r, c) in the left image and to the (r, c + d) in the right image
DSI represents the confidence or likelihood of a particular match.

20. (r, c) slices for different d
Illustration of DSI
(r, c) slices for different d
(c, d) slice for r = 151

21. Definition Match value assigned to element (r, c, d) at iteration n
Initial values computed from SSD or NCC
Inhibition area for element (r, c, d)
Local support area for element (r, c, d)

22. Illustration of Inhibitory and Support Regions

23. Iterative Updating DSI1 2 3 4

24. Explicit Detection of Occlusion
Identify occlusions by examining the magnitude of the converged values in conjunction with the uniqueness constrain

25. Summary of Cooperative Stereo
Prepare a 3D array, (r, c, d): (r, c) for each pixel in the reference image and d for the range of disparity.
Set initial match values using a function of image intensities, such as normalized correlation or SSD.
Iteratively update match values using (4) until the match values converge.
For each pixel (r, c), find the element (r, c, d) with the maximum match value.
If the maximum match value is higher than a threshold, output the disparity d, otherwise, declare a occlusion.

26. MRF Stereo Model Local Evidence function Compatibility function
:Lx1 vector
:LxL matrix

27. Disparity Optimization
Joint probability of MRF:
The disparity optimization step requires choosing an estimator for
MMSE: estimate of the mean of the marginal distribution of
MAP: the labeling of maximize the above joint probability
(1)

28. Equivalence to Energy Minimization
Taking the negative log of equation 1:
In graph cut, equation 2 is expressed as:
Maximizing the probability in equation 1is equivalent to minimizing energy in equation 3.
(2)
(3)

29. Stereo Matching Using Belief Propagation
Belief propagation is an iterative inference algorithm that propagates messages in the Markov network
Message node send to
Message observed node send to
Belief at node
We simplify as , and as

30. Belief Propagation Algorithm
Initialize messages as uniform distribution
Iterative update messages for I = 1:T
Compute belief at each node and output disparity

31. Illustration of BP

32. BP Results

33. Stereo As a Pixel-Labeling Problem
Let P be a set of pixels, L be a label set. The goal is find a labeling f which minimize some energy. For stereo, the labels are disparities.
The classic form of energy function is:

34. Energy Function:
The energy function measures how appropriate a label is for the pixel given the observed data. In stereo, this term corresponds to the match cost or likelihood.
The energy term encodes the prior or smoothness constraint. In stereo, the so called Potts model is used:

35. Two Energy Minimization Algorithm via Graph Cuts
Swap algorithm

36. Two Energy Minimization Algorithm via Graph Cuts
expansion algorithm

37. Moves

38. Graph Cuts Results
Graph Cuts
Belief Propagation

39. Ordering Constraint
If an object a is left on an object b in the left image then object a will also appear to the left of object b in the right image
Ordering constraint…
…and its failure

40. Stereo Correspondences
Left scanline
Right scanline …
Match intensities sequentially between two scanlines

41. Stereo Correspondences
Left scanline
Right scanline …
Left occlusion
Match
Match
Match
Right occlusion

42. Search Over Correspondences
Left Occluded Pixels
Left scanline
Right scanline
Right occluded Pixels
Three cases:
Sequential – cost of match
Left occluded – cost of no match
Right occluded – cost of no match

43. Standard 3-move Dynamic Programming for Stereo
Left Occluded Pixels
Left scanline
Start
Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint
Right occluded Pixels
Right scanline
End

44. Dynamic Programming
Efficient algorithm for solving sequential decision (optimal path) problems. 1 1 1 1 … 2 2 2 2 3 3 3 3
How many paths through this trellis?

45. Dynamic Programming 1 1 1 States: 2 2 2 3 3 3
Suppose cost can be decomposed into stages:

46. Dynamic Programming 1 1 1 2 2 2 3 3 3
Principle of Optimality for an n-stage assignment problem

47. Dynamic Programming 1 1 1 2 2 2 3 3 3

48. Stereo Matching with Dynamic Programming
Pseudo-code describing how to calculate the optimal match

49. Stereo Matching with Dynamic Programming
Pseudo-code describing how to reconstruct the optimal path

50. Results
Local errors may be propagated along a scan-line and no inter scan-line consistency is enforced.

51. Assumption Behind Segmentation-based Stereo
Depth discontinuity tend to correlate well with color edges
Disparity variation within a segment is small
Approximation the scene with piece-wise planar surfaces

52. Segmentation-based stereo
Plane equation is fitted in each segment based on initial disparity estimation obtained SSD or Correlation
Globe matching criteria: if a depth map is good, warping the reference image to the other view according to this depth will render an image that matches the real view
Optimization by iterative neighborhood depth hypothesizing

53. Hypothesizing neighborhood depth
Correct depth is propagated to reduce fattening effect:

54. Hypothesizing neighborhood depth
Background depth is hypothesized for unmatched region:

55. Result

56. Another Result