Top-Down & Bottom-Up Segmentation Is this a Building or a Horse? Do these edges and contours represent anything? Top-Down & Bottom-Up Segmentation Presented By: Joseph Djugash

What’s wrong with Segmentation from Image Statistics? Is this an object boundary? Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Not all Image Statistics are Helpful! Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

How can Class Information help? Where is the object boundary? Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

The Class can help resolve ambiguities! Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Motivation Bottom-Up segmentation Capture image properties Segmentation based on similarities between image regions How can we capture prior knowledge of a specific object (class)? Answer: Top-Down Segmentation

Class-Specific, Top-Down Segmentation Bottom-Up Class-Specific, Top-Down Segmentation Eran Borenstein and Shimon Ullman

Method Fragments Input Matching Cover

Method Outline Fragment Extraction Fragment Matching Segmentation Figure Ground Label Reliability Value Fragment Matching Individual Correspondences Consistency Reliability Segmentation Optimal Cover

Fragment Extraction Want to find fragments that: Generalize well Are specific to the class Add information that other fragments haven’t already given us Fragment Size varies from 1/50 to 1/7 of object size Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

Fragment Extraction Figure-ground label Manual labeling Learned from relative motion or grey level variability Reliability Value – Class Specific Hit rate: A fixed level of false alarms  is achieved by the criterion: Select the k best fragments according to the Hit rate Strength of Response – Maximal normalized correlation of a fragment i with each image I in C and NC

Method Outline Fragment Extraction Fragment Matching Segmentation Figure Ground Label Reliability Value Fragment Matching Individual Correspondences Consistency Reliability Segmentation Optimal Cover

Fragment Matching – Individual Correspondences Measuring Similarity Region Correlation Normalized Correlation Restrict to pixels with the “figure” label Edge Information Derived from the boundary of the figure-ground label The Similarity Measure:

Fragment Matching – Special Requirements The Difference Input Template Entire Template Figure Part Only Figure & Edge Similarity

Fragment Matching – Consistency To acquire a good global cover of the shape, each local match needs to satisfy the consistency measure. Consistency Measure:

Fragment Matching – Reliability Using more “reliable” (anchor) fragments is likely to increase the chance of finding the optimal cover A fragment’s reliability is evaluated by the likelihood ratio between the hit/detection rate and the false alarm rate Reliable fragments used first to guide the covering process

Fragment Matching – Reliability Problem Areas – Do not exactly follow image discontinuities Easy to identify and more commonly seen fragments used first

Method Outline Fragment Extraction Fragment Matching Segmentation Figure Ground Label Reliability Value Fragment Matching Individual Correspondences Consistency Reliability Segmentation Optimal Cover

Segmentation – The Cover Algorithm The best cover should maximize individual match quality, consistency and reliability Thus the cover score is written: Rewards for match quality and reliability Penalizes for inconsistent overlapping fragments Constant that determines the magnitude of the penalty for insufficient consistency Zero for non-overlapping pairs

Segmentation – The Cover Algorithm Initialize with a sub-window that has the maximal concentration of reliable fragments Similarity of all the reliable fragments is examined at 5 scales at all possible locations Iterative Algorithm: Select a small number (M=15) of good candidate fragments Add to cover a subset of the M fragments that maximally improve the score Remove existing fragments inconsistent with new cover (fragments with cumulative negative score) Guaranteed to converge to a local max – score is bounded and increases each iteration

Results I

Results I (cont.)

Results I (cont.)

Eran Borenstein and Shimon Ullman Top-Down Bottom-Up Learning to Segment Eran Borenstein and Shimon Ullman

Method – Old Fragments Input Matching Cover

Method – Updated

Learning Figure-Ground Segmentation – Degree of Cover Start with over-segmented fragments – each fragment now contains many regions Degree of Cover (ri) Calculated by counting the average number of fragments (from C) overlapping the region Ri The fragment selection method extracts most fragments from the figure region Higher ri  higher likelihood to be “figure” Lower ri  lower likelihood to be “background”

Learning Figure-Ground Segmentation – Degree of Cover Most likely figure region By thresholding the degree of cover, ri, we can choose the figure part to be:

Learning Figure-Ground Segmentation – Border Consistency A fragment often contains multiple edges Determine the boundary that optimally separates figure from the background Fragment hit (Hj={1,n}) – image patches where fragment Fi is detected Border Consistency:

Learning Figure-Ground Segmentation – Border Consistency This approach emphasizes consistent edges (border and interior edges) while diffusing noise edges (background features).

Learning Figure-Ground Segmentation Combining degree of cover and boarder consistency we get the figure part (P) Maximized when P contains the most of the consistent edges Maximized when the boundary between the figure and ground are supported by the consistent edges Fragments detected in an image applies its figure-ground “vote” for all the pixels it covers Li(x,y) = +1 – vote for figure label Li(x,y) = –1 – vote for background label i w(i) Li(x,y) – total votes for pixel (x,y) Reliability of fragment i

Improving Figure-Ground Labeling Fragments that are not consistent with the cover (S) is removed and a new cover (S') is generated Further Refinements: Modify the degree of cover to be the average number of times its pixels cover figure parts With a more accurate degree of cover, individual pixels can be substituted for the sub-regions This new degree of cover can them produce an improved cover This iterative approach converges within 3 iterations

Results II

Results II (cont.)

Results II (cont.)

Results II (cont.)

Bottom-Up Segmentation Top-Down Bottom-Up Bottom-Up Segmentation “Segmentation and Boundary Detection Using Multiscale Intensity Measurements” Eitan Sharon, Achi Brandt, and Ronen Basri

Segmentation by Weighted Aggregation Normalized-cuts measure in graphs Detect segments that optimize a NCut measure Hierarchical Structure Recursively coarsen a graph reflecting similarities between intensities of neighboring points Aggregates of pixels of increasing size are gradually collected to form segments Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape, and boundary integrity

Normalized-Cut Measure Minimize: We normalize the cut, E, by the sum of internal couplings, N. The segments minimizing Gamma (PT) will be considered salient. Note that this normalization (PT) encourages large segments. Detecting the salient segments is equivalent to revealing their indicator functions. (Normalization - we only consider segments smaller than half the graph size) Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Segmentation by Weighted Aggregation Normalized-cuts measure in graphs Detect segments that optimize a NCut measure Hierarchical Structure Recursively coarsen a graph reflecting similarities between intensities of neighboring points Aggregates of pixels of increasing size are gradually collected to form segments Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape, and boundary integrity

Bottom-Up Segmentation Here is an example of such an irregular pyramid of aggregates. How is this done? Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Segmentation by Weighted Aggregation Normalized-cuts measure in graphs Detect segments that optimize a NCut measure Hierarchical Structure Recursively coarsen a graph reflecting similarities between intensities of neighboring points Aggregates of pixels of increasing size are gradually collected to form segments Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape, and boundary integrity

Full Texture – Lion Cub Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Full Texture – Polar Bear Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Full Texture - Zebra Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Benefits of the Hierarchical Structure Able to detect regions that differ by fine as well as coarse properties Accurate detection of individual object boundaries Able to detect regions separated by weak, yet consistent edges By combining intensity difference with measures of boundary integrity across neighboring aggregates

Combining Top-Down and Bottom-Up Segmentation Eran Borenstein, Eitan Sharon and Shimon Ullman

Another step towards the middle Top-Down Bottom-Up

Some Definitions & Constraints Measure of saliency h(i), hi є [0,1) A configuration vector s contains labels si (1/-1) of all the segments (Si) in the tree The label si can be different from its parent’s label s i – Cost function for a given s Defines the weighted edge between Si & Si– Top-down term Bottom-up term

Classification Costs The terminal segments of the tree determine the final classification The top-down term is defined as: The saliency of a segment should restrict its label (based on its parent’s label) The bottom-up term is defined as:

Minimizing the Costs – Information Exchange in a Tree Bottom-up message: Top-down message: Min-cost Label: Cost of si = –1 and s = x Message from si = –1 Cost of si = +1 and s = x Message from si = +1 Minimal Cost if the region was classified as background Minimal Cost if the region was classified as figure Computed at each node – minimal of the values is the selected label of node s in s

Confidence Map Evaluating the confidence of a region: Causes of Uncertainty of Classification Bottom-up uncertainty – regions where there is no salient bottom-up segment matching the top-down classification Top-down uncertainty – regions where the top-down classification is ambiguous (highly variable shape regions) The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation

Results III

Results III (cont.)

Results III (cont.)

Results III (cont.)

Results III (cont.)

Results III (cont.)

Top-Down Bottom-Up Questions?

– Appendix – Why Fragments? Vs. Image fragments make good features especially when training data is limited Image fragments contain more information than wavelets allows for simpler classifiers Information theory framework for feature selection Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

– Appendix – Intermediate complexity Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.