Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04) /04 $20.00 c 2004 IEEE 1 Li Hong and George Chen Advanced System Technology San Diego Lab,STMicroelectronics, Inc. Guan-Yu Liu

Outline 2  Introduction  Overview  Method  A : Color segmentation  B : Disparity plane estimation  C : Disparity plane labeling by graph cuts  Experimental Results  Conclusion  Q & A

Introduction (1/3) 3  stereo algorithms can be categorized into two major classes  The first class is local (window-based) algorithms, where the disparity at a given pixel depends only on intensity values within a finite neighboring window.  The second class is global algorithms, which make explicit smoothness assumptions of the disparity map and solve it through various minimization techniques.

Introduction (2/3) 4  Local methods can easily capture accurate disparity in highly textured regions, however they often tend to produce noisy disparities in textureless regions, blur the disparity discontinuous boundaries and fail at occluded areas.  The stereo matching problem is solved through minimizing this global image similarity energy.

Introduction (3/3) 5  Color segment representation is used to reduce the high solution space and enforce disparity smoothness in homogeneous color regions.  A weighted graph is then constructed in which graph nodes represent image pixels, graph label set and graph edge weights correspond to the defined energy terms.  Energy function  Data term  Smoothness term

Method 6  A : Color segmentation  B : Disparity plane estimate  1. Local matching in pixel domain  2. Initial plane fitting from single segment  3. Refined plane fitting from grouped segments  C : Disparity plane labeling by graph cuts

Method.A (1/1) 7  The approach is built upon the assumption that large disparity discontinuities only occur on the boundaries of homogeneous color segments.  Therefore any color segmentation algorithm that decomposes an image into homogeneous color regions will work.  In this paper, mean-shift color segmentation algorithm [3] is used. [3] D. Comaniciu and P. Meer, “Robust Analysis of Feature Spaces: Color Image Segmentation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp , 1997

Method 8  A : Color segmentation  B : Disparity plane estimate  1. Local matching in pixel domain  2. Initial plane fitting from single segment  3. Refined plane fitting from grouped segments  C : Disparity plane labeling by graph cuts

Method.B1 (1/1) 9  In a standard rectified stereo setup, the correspondence between a pixel (x, y) in the reference image I and a pixel (x’, y’) in the matching image J is given by: x’ = x + d(x, y), y’ = y, where the disparity d(x, y) can take any discrete value from the displacement interval [dmin, dmax].  123

Method.B2 (1/2) 10  A plane is used to model the continuous disparity of each segment, i.e., where are the plane parameters and d is the corresponding disparity of the image pixel (x, y). ( ) is the least square solution of a linear system  123  After the initial plane fitting, an iterative process is adopted to update the plane. In each iteration, the pixel disparity is changed within a given range of the fitted plane and the plane parameters are updated based on the modified disparities accordingly.

Method.B2 (2/2) 11  We detect outliers through a simple crosscheck method. Let pixel (x’, y’) be the correspondence of pixel (x, y)  If d(x, y) ≠ d(x’, y’), we consider pixel (x, y) as an outlier.  Weighted least square scheme is adopted in the iteration process.  123  Very small segments are skipped as they lack sufficient data to provide reliable plane estimations.

Method.B3 (1/4) 12  The purpose is not to find the best plane for each segment but rather extract all possible planes for the image. Therefore, it is crucial to extract a set of disparity planes that accurately represent the scene structure.  Step  1. Measure segment matching cost for each plane in the disparity plane set.  2. Assign each segment the plane ID that gives the minimum matching cost.  3. Group neighboring segments with the same plane ID.  4. Apply the plane fitting process mentioned in method.B2 to each grouped segment.

Method.B3 (2/4) 13  It is natural to compute it as the sum of the matching cost from each single pixel inside the segment, i.e.,  123  where S is a segment, P is a disparity plane, and d =  However, there are several problems associated with this approach.  Occluded pixels would easily bias this segment matching cost.

Method.B3 (3/4) 14  They propose two remedies.  exclude all possible occluded pixels in computing the segment matching cost.  augment the sum of pixel matching cost by the percentage of non- supporting pixels to the disparity plane.  They consider only textured outliers as possibly occluded pixels.

Method.B3 (4/4) 15  Let n be the number of non-occluded pixels in a segment S, and let s be the number of supporting pixels to a disparity plane P in segment S. We define the segment matching cost as follows:  123  where O represents the occluded portion in S.

Method 16  A : Color segmentation  B : Disparity plane estimate  1. Local matching in pixel domain  2. Initial plane fitting from single segment  3. Refined plane fitting from grouped segments  C : Disparity plane labeling by graph cuts

Method.C (1/3) 17  They describe in details the formalization of the stereo matching as an energy minimization problem in the segment domain and its solution, i.e., labeling each segment with its corresponding disparity plane by graph cuts.  Let R be the color segments of the reference image, D be the estimated disparity plane set. The goal is to find a labeling f that assigns each segment S ∈ R a corresponding plane f(S) ∈ D, where f is both piecewise smooth and consistent with the observed data.  123 [11] V. Kolmogorov and R. Zabih, “Computing Visual Correspondence with Occlusions using Graph Cuts,” Proc. Int’l Conf. Computer Vision 2001.

Method.C (2/3) 18  123  123  where S and S’ are neighboring segments, uS,S is propor- tional to the common border length between segment S and S’, if f(S) ≠ f(S’), otherwise 0.

Method.C (3/3) 19  The solution converges usually within 2-3 iterations.

Experimental Results (1/4) 20

Experimental Results (2/4) 21

Experimental Results (3/4) 22

Experimental Results (4/4) 23

Conclusions 24  The segment-based approach works well for images with sharp color discontinuities and slanted disparity surfaces.  The current version of our algorithm will not be able to handle the situation if there are disparity boundaries appearing inside the initial color segments.

Computer Science and Information Technology (ICCSIT), rd IEEE International Conference on Volume: 5 25 National University of Singapore Daolei Wang and Kah Bin Lim Guan-Yu Liu

Flow Chart 26

Initial Disparity (1/2) 27  In our paper, initial disparity is obtained by the local matching approach, which is the method of Sum of the Weighted Absolute intensity Differences (SWAD).  Each pixel is assigned a weight w(i, j, d), the value of which results from the 2D Gaussian function of the pixel's Euclidean distance from the central pixel.  123  Where dg is the pixel's Euclidean distance from the central pixel, Tg is the constant parameter.

Initial Disparity (2/2) 28  123  Where the N(i,j) is 5 x 5 surrounding window at the center of the window(i,j).

Plane Fitting (1/5) 29  123  Where the Uh row of A is [X i, Y i, 1], the Uh element in B is d(X i,Y i ). Here we use Singular Value Decomposition (SVD) for least square solution.  123  Where A + is the pseudoinverse of A, the A + can be computed by SVD. Using psedoinverse A+,which compute through SVD, irrespective of A being singular or not.

Plane Fitting (2/5) 30  The cross checking is adopted to get the reliable pixel and filter out occluded pixels and area of the low texture where disparity estimates tend to be unreliable.  Let the DL is the disparity set from left image to right image and the DR is the disparity set from right image to left image.  123

Plane Fitting (3/5) 31  We build a rule to judge the reliable region or unreliable region, the regularity as follow:  123  Where PI is the ratio between the number of the unreliable pixel in the same segment and the number of the segment's pixel.  If a segment satisfies the above Eq, then the all pixels in the segment are labeled by unreliable.

Plane Fitting (4/5) 32  After the above steps filter outliers, we measure the distance between previous disparity to the computed disparity plane, the rule as follow:  123

Plane Fitting (5/5) 33  For the above steps filter outliers, the process of estimation disparity parameters algorithm is then iterated until  is threshold the convergence value of the iterative (typically 0.99).

Neighboring Segment Merging (1/2) 34  Given two segment regions A and B randomly, the plane equations are given by:  123  Then, we can decide whether the two planes are the same or not from two conditions.  The angle  The distance

Neighboring Segment Merging (2/2) 35  We use Gaussian function in two conditions, so the similarity measures as follow function:  123  If C >, where A is constant threshold, then we consider two regions are the same and we merge two segments.

Energy Function (1/2) 36  123  123, Where woec is the penalty coefficient, is the pixel number of detected occlusions which include unreliable pixels in the s segment.  123 Where S N represents a set of all adjacent segments and S i, S j are neighboring segments, Sdisc(S i, S j ) is a discontinuity penalty that incorporates the common border lengths and the mean color similarity as proposed in [6]. [6] M. Gong, R. Yang, W. Liang, and M. Gong. "A perfonnance study on different cost aggregation approaches used in real-time stereo matching.“ Int. Jour. Computer Vision, 75(2): , 2007.

Energy Function (2/2) 37  The solution converges usually with 2-5 iterations. In addition, it is extremely insensitive to the initial labeling.

Experimental Results (1/2) 38

Experimental Results (1/2) 39

Conclusions 40  The algorithm permits us to obtain the high quality dense disparity map of a scene from its initial disparity estimation.  The good points in this paper have three contributions, namely; robust disparity plane fitting, improving Hierarchical clustering algorithm to merge segment and using graph cuts optimization to the new energy function.  The Mean-shift method is a timeconsuming image segmentation algorithm.

Q & A 41