Cross-Based Local Multipoint Filtering Jiangbo Lu1, Keyang Shi2, Dongbo Min1, Liang Lin2, and Minh N. Do3 1Advanced Digital Sciences Center, 2Sun Yat-Sen University, 3Univ. of Illinois at Urbana-Champaign Computer Vision and Pattern Recognition(CVPR), 2012.

Outline Introduction Related Work Proposed Algorithm Experimental Results Conclusion

Introduction

Background Edge-preserving smoothing filtering: A key component for many computer vision applications Goal : remove noise or fine details the structure/edge should be well preserved Bilateral filter(BF), Guided filter(GF)

Cross-Based Local Multipoint Filtering(CLMF) Objective Present a cross-based framework of performing local multipoint filtering efficiently. Two main steps: 1) multipoint estimation 2) aggregation CLMF-0、CLMF-1 Guided Filter (GF) fixed-sized square window Cross-Based Local Multipoint Filtering(CLMF) adaptive window size

Related work

Cross-based local support decision[19] [19] K. Zhang, J. Lu, and G. Lafruit. Cross-based local stereo matching using orthogonal integral images. IEEE Trans. CSVT, 19(7):1073–1079, July 2009.

Bilateral Filter[15] [15] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Proc. of ICCV, 1998.

Guided Filter[6] [6] K. He, J. Sun, and X. Tang. Guided image filtering. In Proc. of ECCV, 2010.

Guided Filter[6]

Guided Filter[6]

Proposed Algorithm

Definition Z : Filter input I : Guidance image Y : Filter output Yi = Zi - ni Z Z : Filter input I : Guidance image Y : Filter output 𝑝 : estimation point 𝑘 : observation point(support pixel) Ωp : local support region of 𝑝 Wp : square window of a radius r {hp, hp, hp, hp } : cross skeleton Y Yi = aIi + b 1 2 3

Adaptive Scale Selection Decide for each direction an appropriate arm length Cross-based method[19] Running average of the intensity of all the pixels covered by the current span h More robust against the measurement noise [19] : If , =1 Otherwise, =0 p h span h (right arm)

Adaptive Scale Selection Gradient reversal artifact

Generalization of Local Multipoint Filtering Zero-order (order m = 0 ) or first-order polynomial(m=1) model: The model should be biased toward low-order polynomials to avoid over-fitting and gradient increase.

Generalization of Local Multipoint Filtering Zero-order (order m = 0 ) or first-order polynomial(m=1) model: Use “least squares” to fit the data (Similar with GF) : ϵ is a regularization parameter to discourage the choices of large (i≥1)

Generalization of Local Multipoint Filtering Zero-order (order m = 0 ) or first-order polynomial(m=1) model: Solutions: Ω 𝑘 : the number of pixels in Ω 𝑘 𝜇 𝑘 : mean of I in Ω 𝑘 𝜎 𝑘 : variance of I in Ω 𝑘 m=0 2 m=1

Generalization of Local Multipoint Filtering Guided Filter(GF) : multipoint estimates are averaged together CLMF : weighted averaged

Summary & Comparison O(1) time linear regression and aggregation (independent of the window radius r)

Experimental Results

Implementation Raw matching cost[9]: Winner-Take-All / Occlusion detection and filling[14] r = 17, R = 3, τ = 20, and τs = 20 1.Scanline filling : the lowest disparity of the spatially closest nonoccluded pixel 2.Median filter : [9] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On building an accurate stereo matching system on graphics hardware. In Proc. of GPUCV, 2011. [14] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. of CVPR, 2011.

CLMF-1 Ground Truth CLMF-0

Experimental Results Middlebury evaluation Rank:23 Tsukuba

Conclusion

Conclusion Propose a generic framework of performing cross-based local multipoint filtering efficiently CLMF-0 and CLMF-1 find very competitive applications into many computer vision More generalized than GF Cross-based technique is very friendly for GPUs[20] Plan to map the filters onto GPUs for speedup [20] K. Zhang, J. Lu, Q. Yang, G. Lafruit, R. Lauwereins, and L. V. Gool. Real-time and accurate stereo: A scalable approach with bitwise fast voting on CUDA. IEEE Trans. CSVT, 21(7):867–878, July 2011.

Full-Image Guided Filtering for Fast Stereo Matching Qingqing Yang, Dongxiao Li, Member, IEEE, Lianghao Wang, and Ming Zhang IEEE SIGNAL PROCESSING LETTERS, VOL. 20, NO. 3, MARCH 2013

Outline Objective Proposed Algorithm Experimental Results Conclusion

Objective Propose a novel full-image guided filtering method A novel scheme called weight propagation is proposed to compute support weights. Edge-preserving Low-complexity

Proposed Algorithm

Filter Modeling C : filter input C’i : filter output at pixel i Wi.j : weight of pixel pair (i,j) Ni : normalizing constant (p.q)∈Pi,j : adjacent nodes on the path Pi,j Tp.q(I) : propagation function Best path : minimum propagation weight → high complexity Choose horizontal first policy Ω : smoothness parameter

Implementation Two pass model 1)Horizontal direction in separate rows 2)the same way in separate columns Pr

Implementation Horizontal: Pr Horizontal: For an element r in a row, the intermediate sum of weighted value : u- : the left neighbor of u u+ : the right neighbor of u Can be further accelerated by using the two-pass scan paradigm[15] The intermediate results are stored in two temporary arrays.

horizontal path → vertical path Implementation Pr Horizontal: The scan process is a sequential computation of weighted cumulative sum: Simply computed by: AL : the weighted cumulative sums calculated from the left to right horizontal path → vertical path reduce the complexity : 4 multiplication and 8 additions (each element)

Implementation Cost Volume C: Winner-Take-All: Post-processing: [18] S. Birchfield and C. Tomasi, “A pixel dissimilarity measure that is insensitive to image sampling,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 20, no. 4, pp. 401–406, 1998. Implementation Cost Volume C: CBT : BT measure[18] CGD : absolute difference of gradient Winner-Take-All: Post-processing: Cross checking : occlusions / mismatch pixels are filled by the lowest disparity value of the nearest non-occluded pixel Weighted median filter

Comparison Employ as many related pixels as possible Important for cost filtering in large textureless regions Bilateral filter Proposed

Comparison Bilateral filter Proposed

Experimental Results

Experimental Results Core Duo 3.16 GHz CPU 2 GB 800MHz RAM No parallelism technique is utilized. The average runtime for cost-volume filtering : 68 ms (on the Middlebury benchmark data sets) 27 × faster than the approach [13] using guided image filtering (1850 ms). [13] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. of CVPR, 2011.

Experimental Results

Experimental Results

Conclusion

Conclusion The novel weight propagation method ensures support elements are assigned. All elements in the input signal contribute to the filtering approach. Outperforms all local methods on the Middlebury benchmark in terms of both speed and accuracy