A Non-local Cost Aggregation Method for Stereo Matching Qingxiong Yang City University of Hong Kong 2012 IEEE Conference on Computer Vision and Pattern Recognition

Outilne Introduction Related Works Method Experimental Results Conclusion

Introduction _________________________

Introduction Goal : Get fast and accurate disparity map. Solution : Non-local cost aggregation + MST Advantage : Better in low textures region Low complexity

Related Works _________________________

Related Works Matching cost computation Cost (support) aggregation 1 Matching cost computation 2 Cost (support) aggregation 3 Disparity computation and optimization 4 Disparity refinement [21] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision (IJCV), 47:7–42, 2002.

Related Works Local methods Global methods 1=>2=>3 A local support region with winner take all Implicit smoothness Fast but inaccurate. 1(=>2)=>3 Energy minimization process (GC,BP,DP,Cooperative) Per-processing Explicit smoothness Accurate but slow

Comparison (Rank in Middleburry) Real time Non-Real Method 1 - 2 O (no Seg.) Cross-based Aggregation>>Scanline Optimization 3 Mean-shift >> BP>>Self-adapting 4 Mean-shift >> Cooperative Optimization 5 6 Mean-shift>>Color-weighted>>BP 7 (no Seg.) Seed Detection>>Scanline Propagation 8 Mean-shift>>BP 9 Mean-shift(Region-based)>>B-spline 10 Up-sample>>Bilateral Filter>>BP 11 12 (No Seg.)>>Convex Relaxation>>regularization 13 Mean-shift>>Image Warping>>BP

Method _________________________

Method Matching cost computation － Bilateral Filter 1 Matching cost computation － Bilateral Filter 2 Cost aggregation － MST 3 Disparity computation and optimization 4 Disparity refinement 　－ Median Filter

Bilateral Filter Every sample is replaced by a weighted average of its neighbors. These weights reflect two forces How close are the neighbor and the center sample How similar are the neighbor and the center sample Edge-preserving and noise reducing smoothing filter

Bilateral Filter q p

Bilateral Filter Center Sample : p Neighborhood : q

Bilateral Filter Total Distance

Bilateral Filter Original image Gaussian wieght Bilateral wieght

Minimum Spanning Tree Kruskal's Algorithm Scan all edges increasing weight order, if an edge is safe, add it to F. 4 1 2 3 5 6 10 A B C D E F G H I J Orginal Graph PPT By Jonathan Davis

1 4 2 5 3 10 6 A B D C J E F H G I 4 B C 4 2 1 A 4 E F 1 2 D 3 10 5 G 5 6 3 4 I H 3 2 J

Sort Edges (in reality they are placed in a priority queue - not sorted - but sorting them makes the algorithm easier to visualize) 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D 3 10 5 G 5 6 3 4 I H 3 2 J

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Cycle Don’t Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Cycle Don’t Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A

Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

Minimum Spanning Tree Orginal Graph 4 1 2 3 A B C D E F G H I J 4 1 2 5 6 10 A B C D E F G H I J

Cost Computation Cd(p) : matching cost for pixel p at disparity level d : aggregated cost -- σS and σR : constants used to adjust the similarity.

Cost Aggregation on a Tree Structure Weight between p and q w(p, q) = | I(p)-I(q)| = image gradient Distance between p and q D(p, q) = sum of weights of the connected edges Similarity between p and q Aggregated cost ＝＞

Bilateral Filter vs Tree Structure

Cost Aggregation on a MST Claim 1. Let Tr denote a subtree of a node s and r denote the root node of Tr, then the supports node s received from this subtree is the summation of the supports node s received from r and S(s, r) times the supports node r received from its subtrees. Supports r = 𝒗 𝑺(𝒓,𝒗) 𝑪 𝒅 𝒗 Supports s = 𝑺 𝒔,𝒓 𝑪 𝒅 𝒓= 𝒗 𝑺(𝒔,𝒗) 𝑪 𝒅 𝒗 = 𝒗 𝑺(𝒔,𝒓)𝑺(𝒔,𝒗) 𝑪 𝒅 𝒗=𝑺(𝒔,𝒓) 𝒗 𝑺(𝒔,𝒗) 𝑪 𝒅 𝒗 s Tr r

Cost Aggregation on a MST Aggregated cost => , if node v is a leaf node P(vc) denote parent of nodevc

Cost Aggregation on a MST

Cost Aggregation on a MST Aggregated cost =>

Cost Aggregation on a MST Cost aggregation process Aggregate the original matching cost Cd from leaf nodes towards root node using Eqn. (6) Aggregate from root node towards leaf nodes using Eqn. (7) Complexity Each level：2 addition/subtraction + 3 multiplication

Disparity Refinement D：the left disparity map Unstable：occlusion, lack of texture, specularity Median filter overlap

Experimental Results _________________________

Experimental Results Device： a MacBook Air laptop computer with a 1.8 GHz Intel Core i7 CPU and 4 GB memory Parameter： σ = 0.1 (non-local cost aggregation) Source : Middlebury http://vision.middlebury.edu/stereo/ HHI database(book arrival) Microsofy i2i database(Ilkay)

Experimental Results Time： Proposed average runtime ： 90 milliseconds (1.25× slower) Unnormalized box filter average runtime ： 72 milliseconds. Local guided image filter average runtime ： 960 milliseconds [7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011. [24] P. Viola and M. Jones. Robust real-time face detection. International Journal of Computer Vision, volume 57, pages 137–154, 2003.

[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.

Experimental Results

Experimental Results

Experimental Results

Conclusion _________________________

Conclusion Contributions Outperform all local cost aggregation methods both in speed and accuracy. Present a near real-time stereo system with accurate disparity results. Future works Apply to parallel algorithms Refine matching cost estimation