1. Top-Down & Bottom-Up Segmentation
Is this a Building or a Horse?
Do these edges and contours represent anything?
Top-Down & Bottom-Up Segmentation
Presented By:
Joseph Djugash

2. What’s wrong with Segmentation from Image Statistics?
Is this an object boundary?
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

3. Not all Image Statistics are Helpful!
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

4. How can Class Information help?
Where is the object boundary?
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

5. The Class can help resolve ambiguities!
Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

6. Motivation Bottom-Up segmentation
Capture image properties
Segmentation based on similarities between image regions
How can we capture prior knowledge of a specific object (class)?
Answer: Top-Down Segmentation

7. Class-Specific, Top-Down Segmentation
Bottom-Up
Class-Specific, Top-Down Segmentation
Eran Borenstein and Shimon Ullman

8. Method
Fragments
Input
Matching
Cover

9. Method Outline Fragment Extraction Fragment Matching Segmentation
Figure Ground Label
Reliability Value
Fragment Matching
Individual Correspondences
Consistency
Reliability
Segmentation
Optimal Cover

10. Fragment Extraction Want to find fragments that: Generalize well
Are specific to the class
Add information that other fragments haven’t already given us
Fragment Size varies from 1/50 to 1/7 of object size
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

11. Fragment Extraction Figure-ground label
Manual labeling
Learned from relative motion or grey level variability
Reliability Value – Class Specific
Hit rate:
A fixed level of false alarms  is achieved by the criterion:
Select the k best fragments according to the Hit rate
Strength of Response –
Maximal normalized correlation of a fragment i with each image I in C and NC

12. Method Outline Fragment Extraction Fragment Matching Segmentation
Figure Ground Label
Reliability Value
Fragment Matching
Individual Correspondences
Consistency
Reliability
Segmentation
Optimal Cover

13. Fragment Matching – Individual Correspondences
Measuring Similarity
Region Correlation
Normalized Correlation
Restrict to pixels with the “figure” label
Edge Information
Derived from the boundary of the figure-ground label
The Similarity Measure:

14. Fragment Matching – Special Requirements
The Difference
Input Template
Entire Template
Figure Part Only
Figure & Edge Similarity

15. Fragment Matching – Consistency
To acquire a good global cover of the shape, each local match needs to satisfy the consistency measure.
Consistency Measure:

16. Fragment Matching – Reliability
Using more “reliable” (anchor) fragments is likely to increase the chance of finding the optimal cover
A fragment’s reliability is evaluated by the likelihood ratio between the hit/detection rate and the false alarm rate
Reliable fragments used first to guide the covering process

17. Fragment Matching – Reliability
Problem Areas –
Do not exactly follow image discontinuities
Easy to identify and more commonly seen fragments used first

18. Method Outline Fragment Extraction Fragment Matching Segmentation
Figure Ground Label
Reliability Value
Fragment Matching
Individual Correspondences
Consistency
Reliability
Segmentation
Optimal Cover

19. Segmentation – The Cover Algorithm
The best cover should maximize individual match quality, consistency and reliability
Thus the cover score is written:
Rewards for match quality and reliability
Penalizes for inconsistent overlapping fragments
Constant that determines the magnitude of the penalty for insufficient consistency
Zero for non-overlapping pairs

20. Segmentation – The Cover Algorithm
Initialize with a sub-window that has the maximal concentration of reliable fragments
Similarity of all the reliable fragments is examined at 5 scales at all possible locations
Iterative Algorithm:
Select a small number (M=15) of good candidate fragments
Add to cover a subset of the M fragments that maximally improve the score
Remove existing fragments inconsistent with new cover (fragments with cumulative negative score)
Guaranteed to converge to a local max – score is bounded and increases each iteration

21. Results I

22. Results I (cont.)

23. Results I (cont.)

24. Eran Borenstein and Shimon Ullman
Top-Down
Bottom-Up
Learning to Segment
Eran Borenstein and Shimon Ullman

25. Method – Old
Fragments
Input
Matching
Cover

26. Method – Updated

27. Learning Figure-Ground Segmentation – Degree of Cover
Start with over-segmented fragments – each fragment now contains many regions
Degree of Cover (ri)
Calculated by counting the average number of fragments (from C) overlapping the region Ri
The fragment selection method extracts most fragments from the figure region
Higher ri  higher likelihood to be “figure”
Lower ri  lower likelihood to be “background”

28. Learning Figure-Ground Segmentation – Degree of Cover
Most likely figure region
By thresholding the degree of cover, ri, we can choose the figure part to be:

29. Learning Figure-Ground Segmentation – Border Consistency
A fragment often contains multiple edges
Determine the boundary that optimally separates figure from the background
Fragment hit (Hj={1,n}) – image patches where fragment Fi is detected
Border Consistency:

30. Learning Figure-Ground Segmentation – Border Consistency
This approach emphasizes consistent edges (border and interior edges) while diffusing noise edges (background features).

31. Learning Figure-Ground Segmentation
Combining degree of cover and boarder consistency we get the figure part (P)
Maximized when P contains the most of the consistent edges
Maximized when the boundary between the figure and ground are supported by the consistent edges
Fragments detected in an image applies its figure-ground “vote” for all the pixels it covers
Li(x,y) = +1 – vote for figure label
Li(x,y) = –1 – vote for background label
i w(i) Li(x,y) – total votes for pixel (x,y)
Reliability of fragment i

32. Improving Figure-Ground Labeling
Fragments that are not consistent with the cover (S) is removed and a new cover (S') is generated
Further Refinements:
Modify the degree of cover to be the average number of times its pixels cover figure parts
With a more accurate degree of cover, individual pixels can be substituted for the sub-regions
This new degree of cover can them produce an improved cover
This iterative approach converges within 3 iterations

33. Results II

34. Results II (cont.)

35. Results II (cont.)

36. Results II (cont.)

37. Bottom-Up Segmentation
Top-Down
Bottom-Up
Bottom-Up Segmentation
“Segmentation and Boundary Detection Using Multiscale Intensity Measurements”
Eitan Sharon, Achi Brandt, and Ronen Basri

38. Segmentation by Weighted Aggregation
Normalized-cuts measure in graphs
Detect segments that optimize a NCut measure
Hierarchical Structure
Recursively coarsen a graph reflecting similarities between intensities of neighboring points
Aggregates of pixels of increasing size are gradually collected to form segments
Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates
Use multiscale measures of intensity, texture, shape, and boundary integrity

39. Normalized-Cut Measure
Minimize:
We normalize the cut, E, by the sum of internal couplings, N. The segments minimizing Gamma (PT) will be considered salient. Note that this normalization (PT) encourages large segments. Detecting the salient segments is equivalent to revealing their indicator functions.
(Normalization - we only consider segments smaller than half the graph size)
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

40. Segmentation by Weighted Aggregation
Normalized-cuts measure in graphs
Detect segments that optimize a NCut measure
Hierarchical Structure
Recursively coarsen a graph reflecting similarities between intensities of neighboring points
Aggregates of pixels of increasing size are gradually collected to form segments
Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates
Use multiscale measures of intensity, texture, shape, and boundary integrity

41. Bottom-Up Segmentation
Here is an example of such an irregular pyramid of aggregates.
How is this done?
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

42. Segmentation by Weighted Aggregation
Normalized-cuts measure in graphs
Detect segments that optimize a NCut measure
Hierarchical Structure
Recursively coarsen a graph reflecting similarities between intensities of neighboring points
Aggregates of pixels of increasing size are gradually collected to form segments
Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates
Use multiscale measures of intensity, texture, shape, and boundary integrity

43. Full Texture – Lion Cub
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

44. Full Texture – Polar Bear
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

45. Full Texture - Zebra
Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

46. Benefits of the Hierarchical Structure
Able to detect regions that differ by fine as well as coarse properties
Accurate detection of individual object boundaries
Able to detect regions separated by weak, yet consistent edges
By combining intensity difference with measures of boundary integrity across neighboring aggregates

47. Combining Top-Down and Bottom-Up Segmentation
Eran Borenstein, Eitan Sharon and Shimon Ullman

48. Another step towards the middle
Top-Down
Bottom-Up

49. Some Definitions & Constraints
Measure of saliency h(i), hi є [0,1)
A configuration vector s contains labels si (1/-1) of all the segments (Si) in the tree
The label si can be different from its parent’s label s i –
Cost function for a given s
Defines the weighted edge between Si & Si–
Top-down term
Bottom-up term

50. Classification Costs
The terminal segments of the tree determine the final classification
The top-down term is defined as:
The saliency of a segment should restrict its label (based on its parent’s label)
The bottom-up term is defined as:

51. Minimizing the Costs – Information Exchange in a Tree
Bottom-up message:
Top-down message:
Min-cost Label:
Cost of si = –1
and s = x
Message from si = –1
Cost of si = +1
and s = x
Message from si = +1
Minimal Cost if the region was classified as background
Minimal Cost if the region was classified as figure
Computed at each node – minimal of the values is the selected label of node s in s

52. Confidence Map Evaluating the confidence of a region:
Causes of Uncertainty of Classification
Bottom-up uncertainty – regions where there is no salient bottom-up segment matching the top-down classification
Top-down uncertainty – regions where the top-down classification is ambiguous (highly variable shape regions)
The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation

53. Results III

54. Results III (cont.)

55. Results III (cont.)

56. Results III (cont.)

57. Results III (cont.)

58. Results III (cont.)

59. Top-Down
Bottom-Up
Questions?

60. – Appendix – Why Fragments?
Vs.
Image fragments make good features
especially when training data is limited
Image fragments contain more information than wavelets
allows for simpler classifiers
Information theory framework for feature selection
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

61. – Appendix – Intermediate complexity
Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.