Spectral Clustering Eyal David Image Processing seminar May 2008

2 Lecture Outline Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations

3 A Tutorial on Spectral Clustering\Arik Azran

4 Spectral Clustering Example – 2 Spirals Dataset exhibits complex cluster shapes K-means performs very poorly in this space due bias toward dense spherical clusters. K-means performs very poorly in this space due bias toward dense spherical clusters. In the embedded space given by two leading eigenvectors, clusters are trivial to separate. Spectral Clustering - Derek Greene

5 Lecture Outline Motivation Graph overview and construction Graph demo Spectral Clustering Spectral Clustering demo Cool implementation

6 Matthias Hein and Ulrike von Luxburg August 2007

7

8

9

10 Matthias Hein and Ulrike von Luxburg August 2007

11 Matthias Hein and Ulrike von Luxburg August 2007

12 Matthias Hein and Ulrike von Luxburg August 2007

13 Matthias Hein and Ulrike von Luxburg August 2007

14 Matthias Hein and Ulrike von Luxburg August 2007

15 Matthias Hein and Ulrike von Luxburg August 2007

16 Matthias Hein and Ulrike von Luxburg August 2007

17 Matthias Hein and Ulrike von Luxburg August 2007

Demo (Live example)

19 Lecture Outline Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations

20 Matthias Hein and Ulrike von Luxburg August 2007

21 Matthias Hein and Ulrike von Luxburg August 2007

22 Matthias Hein and Ulrike von Luxburg August 2007

23 Matthias Hein and Ulrike von Luxburg August 2007

24 Matthias Hein and Ulrike von Luxburg August 2007

25 Matthias Hein and Ulrike von Luxburg August 2007

26 Matthias Hein and Ulrike von Luxburg August 2007

27 Matthias Hein and Ulrike von Luxburg August 2007

28 Matthias Hein and Ulrike von Luxburg August 2007

29 Eigenvectors & Eigenvalues

30 Matthias Hein and Ulrike von Luxburg August 2007

31 Matthias Hein and Ulrike von Luxburg August 2007

Demo (Live example)

33 Spectral Clustering Algorithm Ng, Jordan, and Weiss Motivation Given a set of points We would like to cluster them into k subsets Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

34 Algorithm Form the affinity matrix Define if Scaling parameter chosen by user Define D a diagonal matrix whose (i,i) element is the sum of As row i Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

35 Algorithm Form the matrix Find, the k largest eigenvectors of L These form the the columns of the new matrix X Note: have reduced dimension from nxn to nxk Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

36 Algorithm Form the matrix Y Renormalize each of Xs rows to have unit length Y Treat each row of Y as a point in Cluster into k clusters via K-means Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

37 Algorithm Final Cluster Assignment Assign point to cluster j iff row i of Y was assigned to cluster j Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

38 Why? If we eventually use K-means, why not just apply K-means to the original data? This method allows us to cluster non-convex regions Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

39 Some Examples

40 Ng et al On Spectral clustering: analysis and algorithm

41 Ng et al On Spectral clustering: analysis and algorithm

42 Ng et al On Spectral clustering: analysis and algorithm

43 Ng et al On Spectral clustering: analysis and algorithm

44 Ng et al On Spectral clustering: analysis and algorithm

45 Ng et al On Spectral clustering: analysis and algorithm

46 Ng et al On Spectral clustering: analysis and algorithm

47 Ng et al On Spectral clustering: analysis and algorithm

48 Users Prerogative Affinity matrix construction Choice of scaling factor Realistically, search over and pick value that gives the tightest clusters Choice of k, the number of clusters Choice of clustering method Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

49 Largest eigenvalues of Cisi/Medline data λ1λ1 λ2λ2 How to select k ? Eigengap: the difference between two consecutive eigenvalues. Most stable clustering is generally given by the value k that maximises the expression Choose k=2 Choose k=2 Spectral Clustering - Derek Greene

50 Matthias Hein and Ulrike von Luxburg August 2007 Recap – The bottom line

51 Summary Spectral clustering can help us in hard clustering problems The technique is simple to understand The solution comes from solving a simple algebra problem which is not hard to implement Great care should be taken in choosing the starting conditions

The End