Window-based Approach For Fast Stereo Correspondence Raj Kumar Gupta, Siu-Yeung Cho IET Computer Vision,

Outline Introduction Related Work Proposed Method Experimental Results Conclusion 2

Introduction Using two correlation windows to improve the performance of the algorithm 3*3 and 9*9 Real-time suitability more than 10 frame/s on CPU in case of 320 × 240-sized image pair with disparity value 16 3

Related Work Local methods are usually base on correlation. Area-based (NCC, SAD, SSD) Feature-based: rely on feature extraction and match local cues (BF, GF) Bigger window size, more information, more blurred. 4

Outline Introduction Related Work Proposed Method Experimental Results Conclusion 5

Flow Chart 6

7

Initial Matching 8 Matching cost computation: SAD Left Right d

Problem in disparity selection 9 a. Determine disparity easily for unique minimum value b. Ambiguous disparity in case of multiple minima c. Matching cost calculated at point (205, 230) of Tsukuba image

Initial Matching: large correlation window Matching cost computation: SAD + penalty Penalty term Disparity computation 10

Problem in disparity selection 11

Initial Matching: small correlation window Only those disparity values that are carried by neighbouring pixels. Matching cost computation without penalty N: the disparity values of the neighbouring pixels. Avoid local minima and speed up 12

Flow Chart 13

Unreliable pixel detection left–right cross-checking 14

Disparity Interpolation Search for pixels with reliable disparity value in its eight neighbouring pixels. Compute similarity of unreliable pixel and its reliable neighbor. 15

Flow Chart 16

Disparity Refinement 17

Outline Introduction Related Work Proposed Method Experimental Results Conclusion 18

Experimental Results Computation time of the proposed algorithm for different window sizes on Tsukuba image. (image size 384 × 288 with 16 disparity labels) 19

20 Percentage error in non-occluded (nocc), whole image (all) and near depth discontinuities (disc) for different window sizes for all four images (Tsukuba, Venus, Teddy and Cones)

Experimental Results a. Without using small correlation window Ws and the disparity refinement step b. Without using the disparity refinement step c. Without using small correlation window Ws d. With all four steps on Tsukuba image 21

Experimental Results 22

23

24

Experimental Results Comparison the performance of the proposed algorithm with other correlation-based algorithms. 25

Experimental Results 26

Reference [24] Gupta, R., Cho, S.-Y.: ‘Real-time stereo matching using adaptive binary window’ (3D Data Processing, Visualization and Transmission, 2010) [25] Zhang, K., Lu, J., Lafruit, G., Lauwereins, R., Gool, L.V.: ‘Real-time accurate stereo with bitwise fast voting on Cuda’. Int. Conf. Computer Vision Workshops, 2009, pp. 540–547 [26] Humenberger, M., Zinner, C., Weber, M., Kubinger, W., Vincze, M.:‘A fast stereo matching algorithm suitable for embedded real-time systems’, Comput. Vis. Image Underst., 2010, 114, (11),pp. 1180–1202 [27] Gong, M., Yang, Y.: ‘Near real-time reliable stereo matching using programmable graphics hardware’. IEEE Conf. Computer Vision and Pattern Recognition, 2005, pp. 924–931 [28] Richardt, C., Orr, D., Davies, I., Criminisi, A., Dodgson, N.: ‘Real-time spatiotemporal stereo matching using the dual-cross-bilateral grid’. European Conf. Computer Vision, 2010, vol. 6313, pp. 510–523 [29] Ambrosch, K., Kubinger, W.: ‘Accurate hardware-based stereo vision’,Comput. Vis. Image Underst., 2010, 114, (11), pp. 1303–

28

29

30

31

32

33

Conclusion A new correlation-based stereo-matching approach. Large window improves at non-textured image regions Small window improves at depth discontinuities The CPU implementation computes at a speed of more than 10 frame/s. Easily implemented on GPU. The proposed method can be used in real-time applications to reconstruct the 3D structures with great accuracy at object boundaries. 34

Codebook based Stereo Matching for Natural User Interface Sung-il Kang and Hyunki Hong 2013 IEEE International Conference on Consumer Electronics (ICCE) 35

Outline Introduction Proposed Method Experimental Results Conclusion 36

I ntrodu ction Interactive user interface has been one of the major topics in consumer electronics. Gesture based user interface Interactive smart TV, Nintendo Wii, Sony PlayStation3 Move, and Microsoft Kinect. Propose a stereo system implemented on GPGPU for real-time performance. Employ codebook to solve occlusion. 37

Flow chart 38

Proposed Method Pre-processing Laplace od Gaussian (LoG) filter for alleviating the lighting effects. Cost initialization AD+Census [6] 39 [6] X. Mei, X. Sun, M. Zhou, H. Wang, and X. Zhang, “On building an accurate stereo matchng system on graphics hardware,” Proc. of GPUCV, pp ,

Proposed Method Cost aggregation [6] Cross-based aggregation Color similarity and the length constraint Refinement [6,7] Left-right consistency check Iterative region voting Sub-pixel enhancement 40 d [7] Q. Yang, C. Engels, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color- weighted correlation, hierarchical belief propagation and occlusion handling,” IEEE Transactions on PAMI, 2009.

Proposed Method 41 Occlusion? Find codeword Update codeword Codeword? Yes No Yes No Add a new codeword

Experimental Results 42 Device: Intel Quad 2.66GHz with Nvidia GTX460. Stereo images are captured by a Bumblebee 3 from Point Grey Inc. Time: 80~110ms/frame Stereo matching is implemented on GPU. The codebook generation and its evaluation is on CPU.

Experimental Results 43

Experimental Results 44 [8] K. J. Yoon and I. S. Kweon, “Adaptive support-weight approach for correspondence search,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, pp , [9] C. Richardt, D Orr, I Davies, and A Criminisi, “Real-time spatiotemporal stereo matching using the dual-cross- bilateral grid,” Proc. of ECCV, 2010.

Conclusion Propose a stereo system implemented on GPGPU for real-time performance. Good performance at static background Only. 45