1. 數位三維視訊 楊 家 輝 Jar-Ferr Yang 電腦與通信工程研究所 電機工程學系 國立成功大學 Institute of Computer and Communication Engineering Department of Electrical Engineering National Cheng Kung University, Tainan, Taiwan Stereo Matching for Depth Estimation Digital 3D Video: Chapter 4

2. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-2 What is Stereo Matching? Department of Electrical Engineering, Institute of Computer and Communication Engineering Simple Concept of Stereo Matching : from two camera images to estimate disparity (depth) map with: Same object with same disparity Segmentation Calculate correspond pixels similarity (color & geographic distance) Occlusion handling Refinement

3. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-3 Relationship of Stereo Cameras 3 : Baseline : Depth : Focal length Left view Right view

4. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-4 Relationship Between Depth and Disparity 4 Right view Left camera Right camera Left view : Baseline : Depth : Focal length

5. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-5 Related Work Global Approaches –Energy minimization process (graphic cut, Brief Propagation, Dynamic Programming, Cooperative) –Per-processing –Accurate but slow Local Approaches –A local support region with winner-take-all decision –Fixed or Adaptive Support Weight –Fast but inaccurate.

6. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-6 Concept of Stereo Matching 6 Boundaries ambiguities Seimi-global stereo matching Left image Right image Depth map

7. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-7 An Example of Stereo Matching 7 Raw Matching Cost Cost Aggregation Left ImageRight Image Winner-takes-all Disparity Map Refinement

8. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-8 Raw Matching Cost (Census) Census Features: : bit-wise catenation : intensity of central pixel p : intensity of surrounding pixel q corresponding to pixel p in square N(p) : square window : auxiliary function 127 129 126128129 127131129 001 01281 011 129130129 128129 130129128 111 01291 110 Left Window Right Window Census Raw Matching Cost: Hamming Distance where Hamming denotes the number of positions at which the corresponding bits are different [2]. = 4

9. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-9 Cost Aggregation: Cross-based Method 9 pq : the maximum arm length : indicator function : color threshold : RGB color channel : the intensity of color band : the left, right, up and bottom arm length of the cross. : the number of support pixels in

10. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-10 Winner-takes-all (WTA) 10 : the disparity range

11. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-11 General Stereo Matching Techniques Stereo matching techniques could compute the depth from two or more images captured by different location cameras. Involving Three Primitive Problems: –Calibrating camera positions. –Finding all corresponding points (hardest part) –Computing depth or surfaces. It is a difficulty problem for general stereo matching algorithms

12. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-12 Stereo Vision Triangulate on two images of the same point to recover the depth: –Feature matching across views –Calibrated cameras LeftRight baseline Matching correlation windows across scan lines depth

13. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-13 Pixel Matching For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This leaves too much ambiguity, so: Improvement: match windows

14. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-14 Correspondence Using Correlation LeftRight SSD error disparity LeftRight scanline

15. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-15 Sum of Squared (Pixel) Differences LeftRight

16. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-16 Image Normalization Even when the cameras are identical models, there can be differences in gain and sensitivity. For these reason and more, it is a good idea to normalize the pixels in each window:

17. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-17 Block Images as Vectors LeftRight row 1 row 2 row 3 “Unwrap” image to form vector, using raster scan order Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length. row 2

18. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-18 Image Metrics Sum of Squared Differences (Normalized) Normalized Correlation

19. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-19 Window Size W = 3W = 20 Effect of window size Some approaches have been developed to use an adaptive window size (try multiple sizes and select best match)

20. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-20 Stereo Testing and Comparisons Ground truthScene D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two- Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp. 7-42.

21. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-21 Scharstein and Szeliski Stereo Testing and Comparisons

22. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-22 Results with Window Correlation Window-based matching (best window size) Ground truth

23. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-23 Results with Better Method State of the art : Graph cutsGround truth

24. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-24 Stereo Correspondences …… Left scanlineRight scanline

25. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-25 Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

26. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-26 Search Over Correspondences Three cases: –Sequential – add cost of match (small if intensities agree) –Occluded – add cost of no match (large cost) –Disoccluded – add cost of no match (large cost) Left scanline Right scanline Occluded Pixels Disoccluded Pixels

27. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-27 Stereo Matching by Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

28. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-28 Dynamic Programming Efficient algorithm for solving sequential decision (optimal path) problems. 1 2 3 1 2 3 1 2 3 1 2 3 … How many paths through this trellis?

29. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-29 Dynamic Programming 1 2 3 1 2 3 1 2 3 Suppose cost can be decomposed into stages: States:

30. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-30 Dynamic Programming 1 2 3 1 2 3 1 2 3 Principle of Optimality for an n-stage assignment problem:

31. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-31 Dynamic Programming 1 2 3 1 2 3 1 2 3

32. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-32 Stereo Matching by Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

33. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-33 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching by Dynamic Programming

34. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-34 Scan across grid computing optimal cost for each node given its upper- left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching by Dynamic Programming

35. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-35 Simulation Results

36. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-36 Segmentation-based Stereo

37. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-37 Another Example

38. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-38 Results Using a Good Technique Right ImageLeft ImageDisparity

39. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-39 View Interpolation

40. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-40 Computing Correspondence Another approach is to match edges rather than windows of pixels: Which method is better? –Edges tend to fail in dense texture (outdoors) –Correlation tends to fail in smooth featureless areas

41. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-41 Summary of Stereo Matching Constraints: –Geometry, epipolar constraint. –Photometric: Brightness constancy, only partly true. –Ordering: only partly true. –Smoothness of objects: only partly true. Algorithms: –What you compare: points, regions, features? How you optimize: –Local greedy matches. –1D search. –2D search.

42. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-42 Local Approaches Flow Diagram of Local Methods

43. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-43 Edge Preserving filter: Remove noise and preserve structure/edge, like object consideration. Adaptive Support Weight Bilateral filter(BF) Guided filter(GF) Geodesic diffusion Arbitrary Support Region Local Approaches

44. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-44 Edge Preserving filter : Remove noise and preserve structure/edge, like object consideration. Adaptive Support Weight Bilateral filter(BF) Guided filter(GF) Geodesic diffusion Arbitrary Support Region Local Approaches

45. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-45 Local Stereo Matching Goal : To get a considerable high quality stereo matching but with low complexity and low memory, which is good for real-time applications Successive Weighted Summation (SWS) –Constant time filtering –Weighted aggregation

46. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-46 Cost Computation SAD = Sum of Absolute Differences What is Census? What is Hamming distance?

47. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-47 Census Transform 11000 11000 11X00 00011 11111 121130263139 109115334030 98102786745 476732170198 398699159210 110001100011000001111111 Census transform window :

48. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-48 Hamming Distance of Two Censuses Left image Right image Hamming Distance = 3 110001100011000001111111 111001100101000001111111 XOR 001000000110000000000000

49. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-49 Cost Aggregation

50. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-50 Cost Aggregation

51. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-51 (b)Horizontal effective weights (c)Vertical effective weights(d) 2D effective weights Cost Aggregation

52. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-52 (a) AW (b) Geodesic support (c) Arbitrary support region (d) Census with SAD Comparison With Other Methods

53. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-53 Refinement Using cross-check to detect reliable and occluded region detection ф is a constant (set to 0.1 throughout experiments)

54. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-54 Simulation Results (a) Linear mapping function for reliable pixels based on disparities (b) The resultant map for the left image

55. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-55 Disparity Variations 55 Before  After 0 1.15 1 1.30 2 1.45 3 1.60 4 1.75 5 1.90 6 2.05 7 2.20 8 2.35 9 2.50 10 2.65 11 2.80 12 2.95 13 3.10 14 3.25 15 3.40 16 3.55 17 3.70 18 3.85 19 4 20 4.15 21 4.30 22 4.45 23 4.60 24 4.75 25 4.90 26 5.05 27 5.20 28 5.35 29 5.50 30 5.65 31 5.80 32 5.95 33 6.10 34 6.25 35 6.40 36 6.55 37 6.70 38 6.85 39 7 40 7.15 41 7.30 42 7.45 43 7.60 44 7.75 45 7.90 46 8.05 47 8.20 48 8.35 49 8.50 50 8.65 51 8.80 52 8.95 53 9.10 54 9.25 55 9.40 56 9.55 57 9.70 58 9.85 59 10

56. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-56 (c) Detection of occluded and un-reliable regions (b) Without occlusion handling, bright regions correspond to small disparities Simulation Results

57. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-57 (c) Proposed occlusion handling (b) occlusion handling with no background favoring Simulation Results

58. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-58 Simulation Results

59. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-59 Selection of Weighting Parameter

60. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-60 Experimental Results

61. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-61 Experimental Results

62. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-62 Experimental Results Proposed = SAD + Census

63. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-63 Experimental Results Proposed method is the fastest method without any special hardware implementation among Top-10 local methods of the Middlebury test bench

64. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-64 Proposed O(1) AW Guided filter Geodesic support Arbitrary shaped cross filter Experimental Results

65. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-65 Experimental Results

66. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-66 Computational Time Analyses

67. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-67 Error Analyses

68. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-68 Full-Image Guided Filtering Full-ImageProposed InitializationAD + GradientSAD + Census Aggregation Refinement1.Cross checking (lowest disparity) 2.Weighted median filter 1. Cross checking (normalized disparity) 2. Median filter (background handling) Qingqing Yang, Dongxiao Li, Lianghao Wang, and Ming Zhang, “Full-Image Guided Filtering for Fast Stereo Matching”, Signal Processing Letters, IEEE

69. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-69 Comparison with Full-Image

70. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-70 Results Achieved by Full-Image

71. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-71 Full-Image Results SAD+ Census Results Ground Truth Proposed Versus Full-Image

72. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-72 Full-Image Results Ground Truth SAD+ Census Results Proposed Versus Full-Image

73. National Cheng Kung University, Tainan, Taiwan Department of Electrical Engineering, Institute of Computer and Communication Engineering 1-73 Conclusion