1. Lecture 6 Image Segmentation
Slides by:
David A. Forsyth
Clark F. Olson
Steven M. Seitz
Linda G. Shapiro

2. Segmentation and grouping
Motivation: obtain a compact representation for an image (or set of edge pixels or motion sequence, etc.)
Should support application.
Much work has been done, but broad theory is absent at present.

3. Image segmentation
Collecting edge pixels into curves, lines, etc. is one form of segmentation.
Another form of segmentation is grouping (or clustering) image pixels into coherent regions.
Collect together pixels that “belong together”
Segments are usually disjoint
Segments are often connected
Union of segments should cover image

4. Segmentation
How do we decide what pixels belong together?

5. Segmentation cues
What are properties of pixels (or other objects) that “belong together?”
Some criteria that tend to cause tokens to be grouped.

6. Segmentation cues
More such

7. Segmentation as thresholding
The simplest way to segment a (grey-level) image is to apply a threshold at some brightness value.
How do we know what value to use for thresholding?

8. Thresholding
A histogram can give us information about the distribution of pixels.
Bimodal distributions are best for this technique.
Modes are not always easy to determine.

9. Where do we threshold this distribution?
Thresholding
Most images are not straightforward to threshold.
Where do we threshold this distribution?

10. Thresholding application
Thresholding works best in controlled situations. For example, it is sometimes useful in medical imaging.

11. Segmentation as clustering
Goal: cluster together pixels, that belong together.
Agglomerative clustering:
start with every pixel as separate cluster
combine closest two clusters
repeat
Divisive clustering:
start with every pixel in one cluster
split cluster along best boundary
Pixel-Pixel distances:
color
texture
proximity
etc.
Cluster-Cluster distances
single-link clustering
- minimum distance
complete-link clustering
- maximum distance
average-link clustering
- average distance

12. Segmentation as clustering
Overall methodology for determining the location and composition of clusters is not straightforward.
If we know the cluster locations, how should we assign points to clusters?
If we know the assignment of points to clusters, how should we assign locations to clusters?

13. K-means clustering Must choose a fixed number of clusters.
Choose cluster centers and point-cluster allocations to minimize error.
Can’t do this by search, because there are too many possible allocations.
Algorithm
fix cluster centers; allocate points to closest cluster
fix allocation; compute best cluster centers
Note the distance doesn’t need to be just proximity.
be careful about scaling

14. K-means clustering How many clusters should you use?
K-means clustering will always converge after some number of iterations.
However, it won’t always be the global minimum of the error measure:

15. K-means clustering
Image
Clusters on intensity
Clusters on color
I gave each pixel the mean intensity or mean color of its cluster --- this is basically just vector quantizing the image intensities/colors. Notice that there is no requirement that clusters be spatially localized and they’re not.
K-means clustering using intensity alone and color alone
(5 clusters, no location information included in the distance)

16. Divisive clustering
In divisive clustering, an image is represented by a graph whose nodes are pixels or small groups of pixels.
The goal is to partition the vertices into disjoint sets so that the similarity within each set is high and
across different sets is low.

17. Minimal cuts
Let G = (V,E) be a graph. Each edge (u,v) has a weight cu,v that represents the similarity between u and v.
Graph G can be broken into 2 disjoint graphs with node sets A and B by removing edges that connect these sets.
Let
One way to segment G is to find the minimal cut.
- Fast algorithms for this exist.

18. Normalized cuts
Using minimal cuts favors small sets of pixels to be divided from a larger group.
“Normalized cuts” can improve this.
cuts are weighted using the “volume” of the set
volume: all weights that are connected to nodes in the set

19. Normalized cuts
Example of divisive clustering using color, texture, and location.
Figure from “Image and video segmentation: the normalized cut framework”,
by Shi and Malik, copyright IEEE, 1998
This is figure caption there explains it all.

20. Segmentation as contour optimization
Another approach to segmentation optimizes a contour such that it lies along a strong boundary in the image.

21. Active contours
These methods start with a particular contour and iteratively improve it until a final state is reached.
Several names:
Active contours
Snakes
Intelligent Scissors
Example from: E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp , 1995

22. Intelligent scissors
The goal is to find path a between two pixels that stays on image edges.
Each pixel is given a cost depending on how “edgy” it is:
Edges have low cost
Non-edge have high cost
Now must find lowest cost path.

23. Intelligent scissors Can be formulated as a graph algorithm:
Each pixel in the image is a node
There is a cost between each pair of adjacent pixels (rather than a cost for each pixel)
Contour is path between two pixels that minimizes the cost between each pair of adjacent pixel in the contour
How do we determine the “shortest” path in the graph?
Can determine all shortest paths for one starting pixel in less than 1 second for reasonable image sizes.

24. Expectation maximization
The EM (expectation maximization) algorithm is common in computer vision and is a popular variation of this method.
Clusters are represented by Gaussian distributions
E step: assign points to clusters
- Assignment is “soft” using probability
M step: determine mean, standard deviation of Gaussian

25. Segmentation with EM 3 clusters 4 clusters
This is figure story in the caption
3 clusters
4 clusters
Figure from “Color and Texture Based Image Segmentation Using EM and Its Application to Content Based Image Retrieval”,S.J. Belongie et al., Proc. Int. Conf. Computer Vision, 1998.