Presentation is loading. Please wait.

Presentation is loading. Please wait.

Top-Down & Bottom-Up Segmentation Presented By: Joseph Djugash Is this a Building or a Horse? Do these edges and contours represent anything?

Similar presentations


Presentation on theme: "Top-Down & Bottom-Up Segmentation Presented By: Joseph Djugash Is this a Building or a Horse? Do these edges and contours represent anything?"— Presentation transcript:

1 Top-Down & Bottom-Up Segmentation Presented By: Joseph Djugash Is this a Building or a Horse? Do these edges and contours represent anything?

2 What’s wrong with Segmentation from Image Statistics? Is this an object boundary? Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

3 Not all Image Statistics are Helpful! Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

4 How can Class Information help? Where is the object boundary? Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

5 The Class can help resolve ambiguities! Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

6 Motivation Bottom-Up segmentation Capture image properties Segmentation based on similarities between image regions How can we capture prior knowledge of a specific object (class)? Answer: Top-Down Segmentation

7 Class-Specific, Top-Down Segmentation Eran Borenstein and Shimon Ullman

8 Method Input Fragments MatchingCover

9 Method Outline Fragment Extraction Figure Ground Label Reliability Value Fragment Matching Individual Correspondences Consistency Reliability Segmentation Optimal Cover

10 Fragment Extraction Want to find fragments that: Generalize well Are specific to the class Add information that other fragments haven’t already given us Fragment Size varies from 1 / 50 to 1 / 7 of object size Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

11 Fragment Extraction Figure-ground label Manual labeling Learned from relative motion or grey level variability Reliability Value – Class Specific Hit rate: A fixed level of false alarms  is achieved by the criterion: Select the k best fragments according to the Hit rate Strength of Response – Maximal normalized correlation of a fragment i with each image I in C and NC

12 Method Outline Fragment Extraction Figure Ground Label Reliability Value Fragment Matching Individual Correspondences Consistency Reliability Segmentation Optimal Cover

13 Fragment Matching – Individual Correspondences Measuring Similarity Region Correlation Normalized Correlation Restrict to pixels with the “figure” label Edge Information Derived from the boundary of the figure-ground label The Similarity Measure:

14 Fragment Matching – Special Requirements The Difference Entire Template Figure Part OnlyFigure & Edge Similarity Input Template

15 Fragment Matching – Consistency To acquire a good global cover of the shape, each local match needs to satisfy the consistency measure. Consistency Measure:

16 Fragment Matching – Reliability Using more “reliable” (anchor) fragments is likely to increase the chance of finding the optimal cover A fragment’s reliability is evaluated by the likelihood ratio between the hit/detection rate and the false alarm rate Reliable fragments used first to guide the covering process

17 Fragment Matching – Reliability Easy to identify and more commonly seen fragments used first Problem Areas – Do not exactly follow image discontinuities

18 Method Outline Fragment Extraction Figure Ground Label Reliability Value Fragment Matching Individual Correspondences Consistency Reliability Segmentation Optimal Cover

19 Segmentation – The Cover Algorithm The best cover should maximize individual match quality, consistency and reliability Thus the cover score is written: Penalizes for inconsistent overlapping fragments Rewards for match quality and reliability Constant that determines the magnitude of the penalty for insufficient consistency Zero for non-overlapping pairs

20 Segmentation – The Cover Algorithm Initialize with a sub-window that has the maximal concentration of reliable fragments Similarity of all the reliable fragments is examined at 5 scales at all possible locations Iterative Algorithm: Select a small number (M=15) of good candidate fragments Add to cover a subset of the M fragments that maximally improve the score Remove existing fragments inconsistent with new cover (fragments with cumulative negative score) Guaranteed to converge to a local max – score is bounded and increases each iteration

21 Results I

22 Results I (cont.)

23

24 Learning to Segment Eran Borenstein and Shimon Ullman

25 Method – Old Input Fragments MatchingCover

26 Method – Updated

27 Learning Figure-Ground Segmentation – Degree of Cover Start with over-segmented fragments – each fragment now contains many regions Degree of Cover (r i ) Calculated by counting the average number of fragments (from C) overlapping the region R i The fragment selection method extracts most fragments from the figure region Higher r i  higher likelihood to be “figure” Lower r i  lower likelihood to be “background”

28 Learning Figure-Ground Segmentation – Degree of Cover By thresholding the degree of cover, r i, we can choose the figure part to be: Most likely figure region

29 Learning Figure-Ground Segmentation – Border Consistency A fragment often contains multiple edges Determine the boundary that optimally separates figure from the background Fragment hit (H j={1,n} ) – image patches where fragment F i is detected Border Consistency:

30 Learning Figure-Ground Segmentation – Border Consistency This approach emphasizes consistent edges (border and interior edges) while diffusing noise edges (background features).

31 Learning Figure-Ground Segmentation Combining degree of cover and boarder consistency we get the figure part (P) Maximized when P contains the most of the consistent edges Maximized when the boundary between the figure and ground are supported by the consistent edges Fragments detected in an image applies its figure-ground “vote” for all the pixels it covers L i (x,y) = +1 – vote for figure label L i (x,y) = –1 – vote for background label  i w(i) L i (x, y) – total votes for pixel (x,y) Reliability of fragment i

32 Improving Figure-Ground Labeling Fragments that are not consistent with the cover (S) is removed and a new cover (S ' ) is generated Further Refinements: Modify the degree of cover to be the average number of times its pixels cover figure parts With a more accurate degree of cover, individual pixels can be substituted for the sub-regions This new degree of cover can them produce an improved cover This iterative approach converges within 3 iterations

33 Results II

34 Results II (cont.)

35

36

37 Bottom-Up Segmentation “Segmentation and Boundary Detection Using Multiscale Intensity Measurements” Eitan Sharon, Achi Brandt, and Ronen Basri

38 Segmentation by Weighted Aggregation Normalized-cuts measure in graphs Detect segments that optimize a NCut measure Hierarchical Structure Recursively coarsen a graph reflecting similarities between intensities of neighboring points Aggregates of pixels of increasing size are gradually collected to form segments Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape, and boundary integrity

39 Normalized-Cut Measure Minimize: Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

40 Segmentation by Weighted Aggregation Normalized-cuts measure in graphs Detect segments that optimize a NCut measure Hierarchical Structure Recursively coarsen a graph reflecting similarities between intensities of neighboring points Aggregates of pixels of increasing size are gradually collected to form segments Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape, and boundary integrity

41 Bottom-Up Segmentation Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

42 Segmentation by Weighted Aggregation Normalized-cuts measure in graphs Detect segments that optimize a NCut measure Hierarchical Structure Recursively coarsen a graph reflecting similarities between intensities of neighboring points Aggregates of pixels of increasing size are gradually collected to form segments Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape, and boundary integrity

43 Full Texture – Lion Cub Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

44 Full Texture – Polar Bear Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

45 Full Texture - Zebra Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

46 Benefits of the Hierarchical Structure Able to detect regions that differ by fine as well as coarse properties Accurate detection of individual object boundaries Able to detect regions separated by weak, yet consistent edges By combining intensity difference with measures of boundary integrity across neighboring aggregates

47 Combining Top-Down and Bottom-Up Segmentation Eran Borenstein, Eitan Sharon and Shimon Ullman

48 Another step towards the middle Bottom-Up Top-Down

49 Some Definitions & Constraints Measure of saliency h(  i ), h i є [0,1) A configuration vector s contains labels s i (1/-1) of all the segments (S i ) in the tree The label s i can be different from its parent’s label s i – Cost function for a given s Top-down termBottom-up term Defines the weighted edge between S i & S i –

50 Classification Costs The terminal segments of the tree determine the final classification The top-down term is defined as: The saliency of a segment should restrict its label (based on its parent’s label) The bottom-up term is defined as:

51 Minimizing the Costs – Information Exchange in a Tree Bottom-up message: Top-down message: Min-cost Label: Cost of s i = –1 and s = x Message from s i = –1 Cost of s i = +1 and s = x Message from s i = +1 Computed at each node – minimal of the values is the selected label of node s in s Minimal Cost if the region was classified as background Minimal Cost if the region was classified as figure

52 Confidence Map Evaluating the confidence of a region: Causes of Uncertainty of Classification Bottom-up uncertainty – regions where there is no salient bottom-up segment matching the top-down classification Top-down uncertainty – regions where the top-down classification is ambiguous (highly variable shape regions) The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation

53 Results III

54 Results III (cont.)

55

56

57

58

59 Questions?

60 – Appendix – Why Fragments? Image fragments make good features especially when training data is limited Image fragments contain more information than wavelets allows for simpler classifiers Information theory framework for feature selection Vs. Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

61 – Appendix – Intermediate complexity Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.


Download ppt "Top-Down & Bottom-Up Segmentation Presented By: Joseph Djugash Is this a Building or a Horse? Do these edges and contours represent anything?"

Similar presentations


Ads by Google