1. The Fiedler Vector and Graph Partitioning
Barbara Ball
Clare Rodgers
College of Charleston
Graduate Math Department
Research Under
Dr. Amy Langville

2. Outline General Field of Data Clustering Motivation Importance
Previous Work
Laplacian Method
Fiedler Vector
Limitations
Handling the Limitations

3. Outline Our Contributions Future Work References Experiments
Sorting eigenvectors
Testing Non-symmetric Matrices
Hypotheses
Implications
Future Work
Non-square matrices
Proofs
References

4. Understanding Graph Theory2 3 4 1 6 5 10 7 8 9
Given this graph, there are no apparent clusters.

5. Understanding Graph Theory7 3 2 4 1 6 5 9 10 8
Although the clusters are now apparent, we need a better method.

6. Finding the Laplacian Matrix
A = adjacency matrix
D = degree matrix
Find the Laplacian matrix, L
L = D - A
Rows sum to zero 3 2 4 1 6 5 7 10 8 9

7. Behind the Scenes of the Laplacian Matrix
Rayleigh Quotient Theorem:
seeks to minimize the off-diagonal elements of the matrix
or minimize the cutset of the edges between the clusters
Clusters apparent
Not easily clustered

8. Behind the Scenes of the Laplacian Matrix
Rayleigh Quotient Theorem Solution:
1=0, the smallest right-hand eigenvalue of the symmetric matrix, L
1 corresponds to the trivial eigenvector
v1= e = [1, 1, …, 1].
Courant-Fischer Theorem:
also based on a symmetric matrix, L, searches for the eigenvector, v2, that is furthest away from e.

9. Using the Laplacian Matrix
v2, gives relation information about the nodes.
This relation is usually decided by separating the values across zero.
A theoretical justification is given by
Miroslav Fiedler. Hence, v2 is called
the Fiedler vector.

10. Using the Fiedler Vector
v2 is used to recursively partition the graph by separating the components into negative and positive values.
Entire Graph: sign(V2)=[-, -, -, +, +, +, -, -, -, +]
Reds: sign(V2)=[-, +, +, +, -, -] 4 6 5 7 2 1 3 9 8 10

11. Problems With Laplacian Method
The Laplacian method requires the use of:
an undirected graph
a structurally symmetric matrix
square matrices
Zero may not always be the best choice for partitioning the eigenvector values of v2 (Gleich)
Recursive algorithms are expensive

12. Current Clustering Method
Monika Henzinger, Director of Google Research in 2003, cited generalizing directed graphs as one of the top six algorithmic challenges in web search engines.

13. How Are These Problems Currently Being Solved?
Forcing symmetry for non-square matrices:
Suppose A is an (ad x term) non-square matrix.
B imposes symmetry on the information:
Example:

14. How Are These Problems Currently Being Solved?
Forcing symmetry in square matrices:
Suppose C represents a directed graph.
D imposes bidirectional information by finding the nearest symmetric matrix:
D = C + CT
Example:

15. How Are These Problems Currently Being Solved?
Graphically Adding Data: 1 1 2 2 3 3

16. How Are These Problems Currently Being Solved?
Graphically Deleting Data: 1 1 2 2 3 3

17. Our Wish:
Use Markov Chains and the subdominant right-hand eigenvector ( Ball-Rodgers vector) to cluster asymmetric matrices or directed graphs.

18. Where Did We Get the Idea?
Stewart, in An Introduction to Numerical Solutions of Markov Chains, suggests the subdominant, right-hand eigenvector (Ball-Rodgers vector) may indicate clustering.

19. The Markov Method

20. Different Matrices and Eigenvectors
A: connectivity matrix
L = D – A : Laplacian matrix
rows sum to 0
P: probability Markov matrix
the rows sum to 1
Q = I – P: transitional rate matrix
Respective Eigenvectors:
2nd largest of A
2nd smallest of A
2nd smallest of L
(Fiedler vector)
2nd largest of P
(Ball-Rodgers vector)
2nd smallest of Q

21. Graph 1 Eigenvector Value Plots4 6 5 7 2 1 3 9 8 10
Second Largest of A
Fiedler Vector . 4 . 4 . 3 . 3 . 2 . 2 . 1 . 1 - . 1 - . 2 - . 1 - . 3 - . 2 - . 4 - . 3 - . 5 9 2 3 8 1 7 1 5 4 6 - . 4 8 1 9 2 7 3 1 5 4 6
Ball-Rodgers Vector
Second Smallest of Q . 4 . 4 . 3 . 3 . 2 . 2 . 1 . 1 - . 1 - . 1 - . 2 - . 2 - . 3 - . 3 - . 4 - . 4 4 5 6 1 3 7 2 9 1 8 1 8 9 2 7 3 1 4 5 6

22. Graph 1 Banding Using Eigenvector Values4 6 5 7 2 1 3 9 8 10
Banded A
Banded L
Reorders just by using the indices of the sorted eigenvector – No Recursion
Banded P
Banded Q

23. Graph 1 Reordering Using Laplacian Method4 6 5 7 2 1 3 9 8 10
Reordered L A

24. Graph 1 Reordering Using Markov Method4 6 5 7 2 1 3 9 8 10 A
Reordered P
Reordered Q

25. Graph 2 Eigenvector Value Plots4 16 12 13 5 1 6 11 3
Graph 2 Eigenvector Value Plots 20 8 7 17 14 18 15 22 9 19 2 21 10 23
Second Largest of A
Fiedler Vector . 1 . 3 . 5 . 2 - . 5 . 1 - . 1 - . 1 5 - . 2 - . 1 - . 2 5 - . 2 - . 3 - . 3 5 - . 3 - . 4 1 2 2 2 2 1 1 5 1 8 2 1 9 1 2 2 3 1 4 3 1 6 5 1 3 1 7 1 1 9 6 4 8 7 1 - . 4 9 4 7 1 8 6 1 1 1 7 1 6 3 2 2 1 9 2 1 1 5 2 2 1 2 3 1 8 1 2 1 4 5 1 3
Second Smallest of Q
Ball-Rodgers Vector . 3 . 4 . 2 . 3 . 1 . 2 . 1 - . 1 - . 2 - . 1 - . 3 - . 2 - . 4 - . 3 9 4 7 1 8 6 1 1 1 7 1 6 3 2 2 1 9 2 1 1 5 1 2 1 2 2 1 4 1 8 2 3 5 1 3 5 1 3 2 3 1 8 1 4 1 2 2 1 2 1 5 2 1 2 1 9 2 3 1 6 1 7 1 1 6 8 1 7 4 9

26. Graph 2 Banding Using Eigenvector Values4 16 12 13 5 1 6 11 3
Graph 2 Banding Using Eigenvector Values 20 8 7 17 14 18 15 22 9 19 2 21 10 23
Banded A
Banded L
Nicely banded, but no apparent blocks.
Banded P
Banded Q

27. Graph 2 Reordering Using Laplacian Method4 16 12 13 5 1 6 11 3
Graph 2 Reordering Using Laplacian Method 20 8 7 17 14 18 15 22 9 19 2 21 10 23
Reordered L A

28. Graph 2 Reordering Using Markov Method4 16 12 13 5 1 6 11 3
Graph 2 Reordering Using Markov Method 20 8 7 17 14 18 15 22 9 19 2 21 10 23
Reordered P A
Reordered Q

29. Directed Graph 1 7 2 1 4 9 3 8 10 5
Although it is directed, the Fiedler vector still works. 6

30. Directed Graph 1 7 2 1 4 9 3 8 10 5 6
v2 =

31. Directed Graph 1 Reordering Using Laplacian Method7
Directed Graph 1 Reordering Using Laplacian Method 2 3 1 9 4 10 8 5 6
Reordered L A

32. Directed Graph 1 Reordering Using Markov Method7
Directed Graph 1 Reordering Using Markov Method 2 3 1 9 4 10 8 5 6
Reordered P A

33. Directed Graph 1-B 7 2 1 4 3 9 8 10 5
Was bi-directional 6

34. Directed Graph 1-B The Laplacian Method no longer works on this graph.
Certain edges must be bi-directional in order to make the matrix irreducible.
Currently, to deal with
this problem, a small
number is added to each
element of the matrix. 7 2
.01 3 1 9 4 10 8 5 6

35. Directed Graph 1-B Reordering Using Markov Method7
Directed Graph 1-B Reordering Using Markov Method 2 3 1 4 10 9 5 6 8
Reordered P A

36. Directed Graph 2 Reordering Using Markov Method
Answer
Directed Graph 2 Reordering Using Markov Method
Reordered P A

37. Directed Graph 3 Reordering Using Markov Method
Answer
Directed Graph 3 Reordering Using Markov Method
Reordered P A

38. Directed Graph 4 Reordering Using Markov Method
Answer
Directed Graph 4 Reordering Using Markov Method
Reordered P
10% anti-block elements A

39. Directed Graph 5 Reordering Using Markovian Method
Answer
Directed Graph 5 Reordering Using Markovian Method
30% anti-block elements
Reordered P A
Only the first partition is shown.

40. Hypothesis Implications
Plotting the eigenvector values gives better estimates of the number of clusters
Sometimes, sorting the eigenvector values clusters the matrix without any type of recursive process.
Using the stochastic matrix P cluster asymmetric matrices or directed graphs
A number other than zero may be used to partition the eigenvector values.
Recursive methods are time-consuming. The eigenvector plot takes virtually no time at all and requires very little programming or storage!
Non-symmetric matrices (or directed graphs) can be clustered without altering data!

41. Future Work Experiments on Large Non-Symmetric Matrices
Non-square matrices
Clustering eigenvector values to avoid recursive programming
Proofs
Questions

42. References
Friedberg, S., Insel, A., and Spence, L. Linear Algebra: Fourth Edition.
Prentice-Hall. Upper Saddle River, New Jersey, 2003.
Gleich, David. Spectral Graph Partitioning and the Laplacian with Matlab. January 16,
Godsil, Chris and Royle, Gordon. Algebraic Graph Theory.
Springer-Verlag New York, Inc. New York
Karypis, George.
Langville, Amy. The Linear Algebra Behind Search Engines. The Mathematical Association of
America – Online. December, 2005.
Mark S. Aldenderfer, Mark S. and Roger K. Blashfield. Cluster Analysis .
Sage University Paper Series: Quantitative Applications in the Social Sciences,1984.
Moler, Cleve B. Numerical Computing with MATLAB. The Society for Industrial
and Applied Mathematics. Philadelphia, 2004.
Roiger, Richard J. and Michael W. Geatz. Data Mining: A Tutorial-Based Primer
Addison-Wesley, 2003.
Vanscellaro, Jessica E. “The Next Big Thing In Searching”
Wall Street Journal. January 24, 2006.

43. References
Zhukov, Leonid. Technical Report: Spectral Clustering of Large Advertiser Datasets Part I.
April 10, 2003.
Learning MATLAB
www-groups.dcs.st-and.ac.uk/~history/Mathematicians/Markov.html

44. Eigenvector Example
back

45. Structurally Symmetric
back

46. Theory Behind the Laplacian
Minimize the edges between the clusters

47. Theory Behind the Laplacian
Minimizing edges between clusters is the same as minimizing off-diagonal elements in the Laplacian matrix.
min pTLp
where pi = {-1, 1} and i is the each node.
p represents the separation of the nodes into positives and negatives.
pTLp = pT(D-A)p = pTDp – pTAp
However, pTDp is the sum across the diagonal, so is is a constant.
Constants do not change the outcome of optimization problems.

48. Theory Behind the Laplacian
min pTAp
This is an integer nonlinear program.
This can be changed to a continuous program by using Lagrange relaxation and allowing p to take any value from –1 to 1. We rename this vector x, and let its magnitude be N. So, xTx=N.
min xTAx - (xTx – N)
This can be rewritten as the Rayleigh Quotient:
min xTAx/xTx = 1

49. Theory Behind the Laplacian
1=0 and corresponds to the trivial eigenvector v1=e
The Courant-Fischer Theorem seeks to find the next best solution by adding an extra constraint of x  e.
This is found to be the subdominant eigenvector v2, known as the Fiedler vector.

50. Theory Behind the Laplacian
Our Questions:
The symmetry requirement is needed for the matrix diagonalization of D. Why is D important since it is irrelevant for a minimization problem?
If diagonalization is important, could SVD be used instead?
future