1. Segmentation using eigenvectors
Papers:
“Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000
“Segmentation using eigenvectors: a unifying view”. Yair Weiss, ICCV 1999.
Presenter: Carlos Vallespi

2. Image Segmentation

3. Image segmentation How do you pick the right segmentation?
Bottom up segmentation:
- Tokens belong together because
they are locally coherent.
Top down segmentation:
- Tokens grouped because
they lie on the same object.

4. “Correct” segmentation
There may not be a single correct answer.
Partitioning is inherently hierarchical.
One approach we will use in this presentation:
“Use the low-level coherence of brightness, color, texture or motion attributes to come up with partitions”

5. Outline Introduction Graph terminology and representation.
“Min cuts” and “Normalized cuts”.
Other segmentation methods using eigenvectors.
Conclusions.

6. Outline Introduction Graph terminology and representation.
“Min cuts” and “Normalized cuts”.
Other segmentation methods using eigenvectors.
Conclusions.

7. Graph-based Image Segmentation
Image (I)
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

8. Graph-based Image Segmentation
Image (I)
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

9. Graph-based Image Segmentation
Image (I)
Eigenvector
X(W)
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

10. Graph-based Image Segmentation
Image (I)
Eigenvector
X(W)
Discretization
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

11. Outline Introduction Graph terminology and representation.
“Min cuts” and “Normalized cuts”.
Other segmentation methods using eigenvectors.
Conclusions.

12. Graph-based Image Segmentation
G = {V,E}
V: graph nodes
E: edges connection nodes
Pixels
Pixel similarity
Slides from Jianbo Shi

13. Graph terminology
Similarity matrix:
Slides from Jianbo Shi

14. the size of W is quadratic
Affinity matrix
N pixels
Similarity of image pixels to selected pixel
Brighter means more similar
M pixels
Warning
the size of W is quadratic
with the number
of parameters!
Reshape
N*M pixels
N*M pixels

15. Graph terminology
Degree of node: … …
Slides from Jianbo Shi

16. Graph terminology
Volume of set:
Slides from Jianbo Shi

17. Graph terminology
Cuts in a graph:
Slides from Jianbo Shi

18. Representation Partition matrix X: Pair-wise similarity matrix W:
segments
pixels
Partition matrix X:
Pair-wise similarity matrix W:
Degree matrix D:
Laplacian matrix L:

19. Pixel similarity functions
Intensity
Distance
Texture

20. Pixel similarity functions
Intensity
here c(x) is a vector of filter outputs.
A natural thing to do is to square the outputs
of a range of different filters
at different scales and orientations,
smooth the result, and rack
these into a vector.
Distance
Texture

21. Definitions
Methods that use the spectrum of the affinity matrix to cluster are known as spectral clustering.
Normalized cuts, Average cuts, Average association make use of the eigenvectors of the affinity matrix.
Why these methods work?

22. Spectral Clustering Data Similarities
* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University

23. Eigenvectors and blocks
Block matrices have block eigenvectors:
Near-block matrices have near-block eigenvectors:
1= 2
2= 2
3= 0
4= 0 1
.71
.71
eigensolver
1= 2.02
2= 2.02
3= -0.02
4= -0.02 1 .2
-.2
.71
.69
.14
-.14
.69
.71
eigensolver
* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University

24. Spectral Space Can put items into blocks by eigenvectors:
Clusters clear regardless of row ordering: e1 1 .2
-.2
.71
.69
.14
-.14
.69
.71 e2 e1 e2 e1 1 .2
-.2
.71
.14
.69
.69
-.14
.71 e2 e1 e2
* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University

25. Outline Introduction Graph terminology and representation.
“Min cuts” and “Normalized cuts”.
Other segmentation methods using eigenvectors.
Conclusions.

26. How do we extract a good cluster?
Simplest idea: we want a vector x giving the association between each element and a cluster
We want elements within this cluster to, on the whole, have strong affinity with one another
We could maximize
But need the constraint
This is an eigenvalue problem - choose the eigenvector of W with largest eigenvalue.

27. Minimum cut Criterion for partition: A Problem! B Cuts with
Weight of cut is directly proportional to the number of edges in the cut. B
Ideal Cut
Cuts with
lesser weight
than the
ideal cut
First proposed by Wu and Leahy

28. Normalized Cut
Normalized cut or balanced cut:
Finds better cut

29. Normalized Cut
Volume of set (or association): A B

30. Normalized Cut Volume of set (or association):
Define normalized cut: “a fraction of the total edge connections to all the nodes in the graph”: A B A B
Define normalized association: “how tightly on average nodes within the cluster are connected to each other” A B

31. y’s values are quantized
Observations(I)
Maximizing Nassoc is the same as minimizing Ncut, since they are related:
How to minimize Ncut?
Transform Ncut equation to a matricial form.
After simplifying:
NP-Hard!
y’s values are quantized
Subject to:
Rayleigh quotient

32. Observations(II)
Instead, relax into the continuous domain by solving generalized eigenvalue system:
Which gives:
Note that so, the first eigenvector is y0=1 with eigenvalue 0.
The second smallest eigenvector is the real valued solution to this problem!!
min

33. Algorithm Define a similarity function between 2 nodes. i.e.:
Compute affinity matrix (W) and degree matrix (D).
Solve
Use the eigenvector with the second smallest eigenvalue to bipartition the graph.
Decide if re-partition current partitions.
Note: since precision requirements are low, W is very sparse and only few eigenvectors are required, the eigenvectors can be extracted very fast using Lanczos algorithm.

34. Discretization
Sometimes there is not a clear threshold to binarize since eigenvectors take on continuous values.
How to choose the splitting point?
Pick a constant value (0, or 0.5).
Pick the median value as splitting point.
Look for the splitting point that has the minimum Ncut value:
Choose n possible splitting points.
Compute Ncut value.
Pick minimum.

35. Use k-eigenvectors Recursive 2-way Ncut is slow.
We can use more eigenvectors to re-partition the graph, however:
Not all eigenvectors are useful for partition (degree of smoothness).
Procedure: compute k-means with a high k. Then follow one of these procedures:
Merge segments that minimize k-way Ncut criterion.
Use the k segments and find the partitions there using exhaustive search.
Compute Q (next slides). e1 1 .2
-.2
.71
.69
.14
-.14
.69
.71 e2 e1 e2

36. Images from Matthew Brand (TR-2002-42)
Toy examples
Images from Matthew Brand (TR )

37. Example (I)
Eigenvectors
Segments

38. Example (II) Original Segments
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

39. Outline Introduction Graph terminology and representation.
“Min cuts” and “Normalized cuts”.
Other segmentation methods using eigenvectors.
Conclusions.

40. Other methods Average association
Use the eigenvector of W associated to the biggest eigenvalue for partitioning.
Tries to maximize:
Has a bias to find tight clusters. Useful for gaussian distributions. A B

41. Other methods Average cut Tries to minimize:
Very similar to normalized cuts.
We cannot ensure that partitions will have a a tight within-group similarity since this equation does not have the nice properties of the equation of normalized cuts.

42. Other methods

43. Other methods Normalized cut Average cut
20 points are randomly distributed from 0.0 to 0.5
12 points are randomly distributed from 0.65 to 1.0
Average association

44. Other methods Scott and Longuet-Higgins (1990).
Data W
First ev
Second ev Q
Scott and Longuet-Higgins (1990).
V contains the first eigenvectors of W.
Normalize V by rows.
Compute Q=VTV
Values close to 1 belong to the same cluster.

45. Other applications Costeira and Kanade (1995).
Data M Q
Costeira and Kanade (1995).
Used to segment points in motion.
Compute M=(XY).
The affinity matrix W is compute as W=MTM. This trick computes the affinity of every pair of points as a inner product.
Compute Q=VTV
Values close to 1 belong to the same cluster.

46. Other applications Face clustering in meetings.
Grab faces from video in real time (use a face detector + face tracker).
Compare all faces using a distance metric (i.e. projection error into representative basis).
Use normalized cuts to find best clustering.

47. Outline Introduction Graph terminology and representation.
“Min cuts” and “Normalized cuts”.
Other segmentation methods using eigenvectors.
Conclusions.

48. Conclusions Good news: Bad news:
Simple and powerful methods to segment images.
Flexible and easy to apply to other clustering problems.
Bad news:
High memory requirements (use sparse matrices).
Very dependant on the scale factor for a specific problem.

49. The End!
Thank you!

50. Images from Matthew Brand (TR-2002-42)
Examples
Spectral Clutering
Images from Matthew Brand (TR )

51. Solve clustering for affinity matrix
Spectral clustering
Makes use of the spectrum of the similarity matrix of the data to cluster the points.
Solve clustering for affinity matrix
w(i,j) distance node i to node j

52. Graph terminology Similarity matrix: Degree of node: Volume of set:
Graph cuts: