Spectral Clustering 指導教授 : 王聖智 S. J. Wang 學生 : 羅介暐 Jie-Wei Luo

Outline Motivation Graph overview Spectral Clustering Another point of view Conclusion

Motivation  K-means performs very poorly in this space due Dataset exhibits complex cluster shapes cluster shapes

K-means

Spectral Clustering Scatter plot of a 2D data set K-means ClusteringSpectral Clustering U. von Luxburg. A tutorial on spectral clustering. Technical report, Max Planck Institute for Biological Cybernetics, Germany, 2007.

Graph overview Graph Partitioning Graph notation Graph Cut Distance and Similarity

Graph partitioning First-graph representation of data Then-graph partitioning In this talk–mainly how to find a good partitioning of a given graph using spectral properties of that graph

Graph notation Always assume that similarities s ij are symmetric, non-negative Then graph is undirected, can be weighted

Graph notation Degree of vertex v i є V v i

Graph Cuts Mincut : min Cut(A 1,A 2 ) However, mincut simply seperates one individual vertex from the rest of the graph Balanced cut Problem: finding an optimal graph (normalized) cut is NP-hard Approximation: spectral graph partitioning

12

13

14

Spectral Clustering Unnormalized graph Laplacian Normalized graph Laplacian Other explanation Example

Spectral clustering - main algorithms Input: Similarity matrix S, number k of clusters to construct Build similarity graph Compute the first k eigenvectors v 1,..., v k of the problem matrix L for unnormalized spectral clustering L rw for normalized spectral clustering Build the matrix V є R n×k with the eigenvectors as columns Interpret the rows of V as new data points Z i є R k Cluster the points Z i with the k-means algorithm in R k

Example L: Laplacian matrix Similarity Graph W: adjacency matrixD: degree matrix

Example Similarity Graph L: Laplacian matrix Double Zero Eigenvalue  Two Connected Components First Two Eigenvectors v1v1 v2v2

Example Similarity Graph First k Eigenvectors  New Clustering Space y1 y2 y3 y4 y5 v1v2 Use k-means clustering in the new space v2v2 v1v1

Unnormalized graph Laplacian Define as L=D-W proof

Unnormalized graph Laplacian proof Relation between spectrum and clusters: Multiplicity of k eigenvalue 0 = number k of connected components A 1,..., A k of the graph. eigenspace is spanned by the characteristic functions 1 A1,..., 1 Ak of those components (so all eigenvecotrs are piecewise constant).

Unnormalized graph Laplacian Interpret s ij = 1 / d(X i, X j ) 2 looks like a discrete version of the standard Laplace operator

Normalized graph Laplacian Define Relation between L sym & L rw L sym L rw Eigenvalueλλ EigenvectorD1/2uD1/2uu

Normalized graph Laplacian Spectral properties similar to L: Positive semi-definite, smallest eigenvalue is 0 Attention: For L rw, eigenspace spanned by 1A i (piecewise const.) but for Lsym, eigenspace spanned by D 1/2 1A i (not piecewise const).

Random walk explanations General observation: Random walk on the graph has transition matrix P = D −1 W. note that L rw = I − P Specific observation about Ncut : define P(A|B) is the probability to jump from B to A if we assume that the random walk starts in the stationary distribution. Then: Ncut(A, B) = P(A|B) + P(B|A) Interpretation: Spectral clustering tries to construct groups such that a random walk stays as long as possible within the same group

Possible Explanations

Example-2 In the embedded space given by two leading eigenvectors, clusters are trivial to separate.

Example-3 In the embedded space given by three leading eigenvectors, clusters are trivial to separate.

Another point of view

Connections

PCA Linear combination the original data X i to now variable Z i

Rank reduce comparison between PCA & Laplacian Eigenmap PCA is linear combination to reduce dimension, though PCA minimize the Reconstruction error, but it’s not helpful to cluster groups. Spectral clustering is nonlinear reducing dimension which is helpful to cluster, however it’s doesn’t actually have a “rank reduce function” apply to new data, while PCA have it.

Conclusion Why is spectral clustering useful? Does not make strong assumptions on cluster shape Is simple to implement (solving an eigenproblem) Spectral clustering objective does not have local optima Has several different derivations Successful in many applications What are potential problems? Can be sensitive to choice of parameters (k in kNN-graph). Computational expensive on large non-sparse graphs