Local Stereo Matching Using Motion Cue and Modified Census in Video Disparity Estimation Zucheul Lee, Ramsin Khoshabeh, Jason Juang and Truong Q. Nguyen (UCSD) 20th European Signal Processing Conference (EUSIPCO 2012) 1

Outline Introduction Framework Proposed Algorithm Experimental Results Conclusion 2

Introduction 3

Background The disparity estimation has been thoroughly studied Focus strictly on images Video disparity estimation: (1) Lack of video datasets with ground-truth disparity maps (2) Temporal inconsistency problems flickering resulting from simply applying image-based algorithms to video 4

Background Fundamental attributes that group objects together locally: Proximity Similarity Motion The objects grouped by these attributes are most likely to have the same depth. 5 Image disparity estimation - Important for accurate depth estimation near edges of moving objects

Objective Propose a more accurate and noise tolerant method for video disparity estimation More accurate than other methods on edges and in flat (textureless) areas Using: Motion cues (edges) Modified census transform (flat areas) Spatio-temporal consistency (refinement) 6

Related Work Adaptive Weight [6] Cost-volume filtering [7] Guided filter Spatio-Temporal Consistency [3] 7 [7] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, “Fast Cost-Volume Filtering for Visual Correspondence and Beyond,” in Proc.IEEE Intl. Conf. Comput. Vis. Pattern Recognit. (CVPR), pp ,2011. [3] R. Khoshabeh, S. H. Chan, and T. Q. Nguyen, “Spatio-Temporal Consistency in Video Disparity Estimation,” ICASSP, pp , 2011 [6] K.-J. Yoon and I.-S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Anal. Mach. Intell., vol.28, no. 4, pp , Do not provide a reliable solution for disparity estimation in textureless (flat) areas

Framework 8

9 Support Weight Correlated color and motion Matching Cost Modified Census Transform Cost Aggregation and Disparity Computation Refinement Spatio-temporal Consistency Method

Proposed Algorithm 10

Support Weight Using Correlated Color and Motion 11

Support Weight Using Correlated Color and Motion Let and be the color coordinates of pixel c and neighbor pixel q in the CIELab color space Color difference: Let and be the flow vectors [10] of pixel c and neighbor pixel q Truncated motion difference: τ : truncation value 12 [10] D. Sun, S. Roth, M.J. Black, “Secrets of Optical Flow Estimation and Their Principles,” CVPR, pp , 2010.

13 Benefits of a Motion Cue The “car” video frames (480x disparity levels): ProximityProximity + SimilarityProximity + Similarity + Motion

Modified Census Transform Difficult in finding the correct correspondences in flat areas. Due to the fact that the census matching cost is extremely sensitive to image noise since all pixels in flat areas have a similar intensity. Three moded census transform with a noise buffer 14 Problem: Solution:

Modified Census Transform Using two bits to implement three modes α:noise buffer threshold 15 Set 10 if (neighbor pixel intensity) - (center pixel intensity) > α Set 01 if (neighbor pixel intensity) - (center pixel intensity) < α Set 00 otherwise Intensity value 0~50 α = 0 Intensity value 50~100 α = 1 Intensity value 100~150 α = 2 Intensity value 150~200 α = 3 Intensity value 200~255 α = 4

Modified Census Transform 16

Aggregation and disparity Computation Aggregated matching cost: Winner-take-all (WTA): 17

Aggregation and disparity Computation 18 Left viewOriginal census Modified census (without intensity difference) Modified census

Spatio-temporal Consistency [3] Simply applying image-based algorithms to individual frames temporally inconsistent (even the best methods) Consider the sequence of disparity maps as a space-time volume A three-dimensional function f(x,y,t) with (x,y) : spatial coordinates t : temporal coordinate Piecewise smooth solution: has less temporal noise preserves the disparity information as much as possible 19 Problem: Solution:

Spatio-temporal Consistency [3] l 1 – minimization problem: f : unknown disparity map g : initial disparity map from the previous step D : forward difference operator 20

Spatio-temporal Consistency [3] l 1 – minimization problem: f : unknown disparity map f(x,y,t) : Each frame of the video : M rows, N columns Total: K frames Stack the entries of f(x,y,z) into a column vector of size MNK x 1 21 x (M rows) y (N columns) t (K frames)

Spatio-temporal Consistency [3] l 1 – minimization problem: D : forward difference operator : parameters(constants) 22

Spatio-temporal Consistency [3] Solve : 23 [1]S. H. Chan, R. Khoshabeh, K. B. Gibson, P. E. Gill, and T. Q. Nguyen, “An augmented lagrangian method for total variation video restoration,” in ICASSP, May 2011 Introduce two intermediate variables: r = f - g u = Df Unconstrained problem → Constrained problemAugmented Lagrangian[1] Alternation Direction Method(ADM) Solve sub-problem : f,u,r iteratively

Experimental Results 24

Experimental Results 25 [14] C. Richardt, D. Orr, I. Davies, A. Criminisi, and N. A. Dodgson, “Real-time Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral Grid,” ECCV, 2010.

Experimental Results Jamie1 from Microsoft i2i database 26

Experimental Results Ilkay from Microsoft i2i database 27

Experimental Results Tunnel 28

Experimental Results Performance comparison of methods 29 The average percentage of bad pixels (threshold of 1)

Experimental Results 19s to compute the disparity map Can be adopted into a real-time application (by using GPU) Refinement using the TV method [3] reduces errors in the background (spatial noise and temporal inconsistencies) 30

Experimental Results Spatio-Temporal Consistency [3] 31

Experimental Results Spatio-Temporal Consistency [3] 32

Experimental Results Spatio-Temporal Consistency [3] 33

Conclusion 34

Conclusion Propose an accurate local stereo matching method for video disparity estimation Motion cue To obtain more accurate support weight Modified census transform To obtain more reliable raw matching costs in flat areas Spatio-temporal volume Improve spatial and temporal consistency It presents the probability for directly extending current image-based disparity algorithms to the video domain 35