Heather Dunlop 16-721: Advanced Perception January 25, 2006 Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cues by David R. Martin, Charless C. Fowlkes, Jitendra Malik Heather Dunlop 16-721: Advanced Perception January 25, 2006

What is a Boundary? Canny Human Martin, 2002 “A boundary is a contour in the image plane that represents a change in pixel ownership from one object or surface to another” Edges are not boundaries

Dataset “You will be presented a photographic image. Divide the image into some number of segments, where the segments represent ‘things’ or ‘parts of things’ in the scene. The number of segments is up to you, as it depends on the image. Something between 2 and 30 is likely to be appropriate. It is important that all of the segments have approximately equal importance.”

Dataset Database of over 1000 images and 5-10 segmentations for each Thicker lines indicate more common boundary choice Martin, 2002

Boundaries Non-boundaries Boundaries Intensity Brightness Color Brightness on its own is not sufficient Color Texture Martin, 2002

Method Goal: learn the probability of a boundary, Pb(x,y,θ) Image Optimized Cues Boundary Strength Brightness Color Texture Benchmark Human Segmentations Cue Combination Model Goal: “use features extracted from such an image patch to estimate the posterior probability of a boundary passing through the center point” Use cues such as intensity, brightness, color and texture to get a measure of boundary strength How to combine cues? It’s a supervised learning problem. Learn an optimal local boundary model from labeled images Approach: look at each pixel for local discontinuities in several feature channels, over a range of orientations and scales Martin, 2002

Image Features CIE L*a*b* color space (luminance, red-green, yellow-blue) Oriented Energy: fe: Gaussian second derivative fo: Its Hilbert transform Brightness L* distribution Color a* and b* distributions (joint or marginal) Texture “In natural images, brightness edges are more than simple steps. Phenomena such as specularities, mutual illumination, and shading result in composite intensity profiles consisting of steps, peaks and roofs.” even and odd symmetric filters has maximum response for contours at orientation theta -minimal accuracy difference between joint and marginal distributions -joint is much more computation intensive, so use marginal

Texture Convolve with a filter bank: Gaussian second derivative Its Hilbert transform Difference of Gaussians Filter responses give a measure of texture Each pixel is associated with a vector of 13 filter responses centered at that pixel

Other Filter Banks Leung-Malik filter set: Schmid filter set: Maximum Response 8 filter set: MR8: take maximum over orientations

Textons Convolve image with filter bank Cluster filter responses to form textons Adapted from Martin, 2002 and Varma, Zisserman, 2005

Texton Distribution Assign each pixel to nearest texton Form distribution of textons Adapted from Martin, 2002 and Varma, Zisserman, 2005

Gradient-based Features Brightness (BG), color (CG), texture (TG) gradients Half-disc regions described by histograms Compare distributions with χ2 statistic  r (x,y) “At at location (x,y) in the image, draw a circle of radius r, and divide it along the diameter at orientation theta. The gradient function G(x,y,theta,r) compares the contents of the two resulting disc halves. A large difference between the disc halves indicates a discontinuity in the image along the disc’s diameter.” 8 orientations, 3 scales

Texture Gradient Texton distribution in two half circles Martin, 2002

Localization Tightly localize boundaries Reduce noise Coalesce double detections Improve OE and TG features OE OE localized Fit a peak (parabola) TG TG localized Martin, Fowlkes, Malik, 2004

Optimization Texture parameters: type of filter bank scale of filters number of textons universal or image-specific textons Other possible distance/histogram comparison metrics Number of bins for histograms Scale parameter for all cues It was found that a single scale is sufficient for texture Universal vs. image-specific textons: -computational cost approximately equal -accuracy approx equal -optimal number of textons for universal textons is roughly double that for image-specific textons -universal preferable if doing image retrieval and object recognition -image-specific used for convenience L1, L2 norm, chi-square, quadratic form, Earth Mover’s distance

Evaluation Methodology Posterior probability of boundary: Pb(x,y,θ) Evaluation measure: precision recall curve F-measure: “formulate boundary-detection as a classification problem of discriminating non-boundary from boundary pixels” -take maximum over orientations PR curve: “captures the tradeoff between accuracy and noise as the detector threshold varies” “Precision is the fraction of detections that are true positives rather than false positives, while recall is the fraction of true positives that are detected rather than missed. In probabilistic terms, precision is the probability that the detector’s signal is valid, and recall is the probability that the ground truth data was detected.” “Each point on the PR curve is computed from the detector’s output at a particular threshold.” “First, we correspond the machine boundary map separately with each human map in turn. Only those machine boundary pixels that match no human boundary are counted as false positives. The hit rate is simply averaged over the different human, so that to achieve perfect recall the machine boundary map must explain all of the human data.” F-measure: defines a relative cost between precision and recall for a specific application The location of maximum F-measure along the curve provides the optimal detector threshold given alpha. Canny’s goals of boundary detection: high detection rate, single detection, good localization Martin, 2002

Cue Combination Which cues should be used? OE is redundant when other cues are present BG+CG+TG produces best results Martin, 2002

Classifiers Until now, only logistic regression was used Other possible classifiers: Density estimation Classification trees Hierarchical mixtures of experts Support vector machines Logistic regression: fast convergence and reliable svm: prohibitively slow didn’t always produce meaningful results performance of all classifiers is approximately equal Martin, 2002

Result Comparison Alternative methods: Matlab’s Canny edge detector with and without hysteresis Spatially-averaged second moment matrix (2MM) Where’s the human curve come from? “The points marked by a ‘+’ on the plot show the precision and recall of each ground truth human segmentation when compared to other humans. . . . The solid curve shows the F=0.80 curve, representing the frontier of human performance for this task.” Martin, 2002

Results Canny 2MM BG+CG+TG Human Image Martin, 2002 -texture on man’s shirt -no boundary on shoulder Martin, 2002

Results Canny 2MM BG+CG+TG Human Image Martin, 2002 -windows are texture not boundary -underside of boat and on dock Martin, 2002

Results Canny 2MM BG+CG+TG Human Image Martin, 2002

Conclusions Large data set used for testing Texture gradients are a powerful cue Simple linear model sufficient for cue combination Outperforms existing methods An approach that is useful for higher-level algorithms Code is available online: http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/