1. Graph Algorithms in Numerical Linear Algebra: Past, Present, and Future John R. Gilbert MIT and UC Santa Barbara September 28, 2002

2. Outline Past (one extended example) Present (two quick examples) Future (6 or 8 examples)

3. A few themes Paths Locality Eigenvectors Huge data sets Multiple scales

4. PAST

5. Graphs and sparse Gaussian elimination (1961-) 10 1 3 2 4 5 6 7 8 9 1 3 2 4 5 6 7 8 9 G(A) G + (A) [chordal] Cholesky factorization: for j = 1 to n add edges between j’s higher-numbered neighbors Fill: new nonzeros in factor

6. Fill-reducing matrix permutations Theory: approx optimal separators => approx optimal fill and flop count Orderings: nested dissection, minimum degree, hybrids Graph partitioning: spectral, geometric, multilevel

7. Directed graph A is square, unsymmetric, nonzero diagonal Edges from rows to columns Symmetric permutations PAP T 1 2 3 4 7 6 5 AG(A)

8. + Symbolic Gaussian elimination Add fill edge a -> b if there is a path from a to b through lower-numbered vertices. 1 2 3 4 7 6 5 AG (A) L+U

9. 1524736 1 5 2 4 7 3 6 Strongly connected components Symmetric permutation to block triangular form Solve Ax=b by block back substitution Irreducible (strong Hall) diagonal blocks Row and column partitions are independent of choice of nonzero diagonal Find P in linear time by depth-first search 1 2 3 4 7 6 5 PAP T G(A)

10. Sparse-sparse triangular solve 15234 = G(L T ) 1 2 3 4 5 Lxb 1.Symbolic: –Predict structure of x by depth-first search from nonzeros of b 2.Numeric: –Compute values of x in topological order Time = O(flops)

11. Column intersection graph G  (A) = G(A T A) if no cancellation (otherwise  ) Permuting the rows of A does not change G  (A) 15234 1 2 3 4 5 15234 1 5 2 3 4 AG  (A)ATAATA

12. Filled column intersection graph G  (A) = symbolic Cholesky factor of A T A In PA=LU, G(U)  G  (A) and G(L)  G  (A) Tighter bound on L from symbolic QR Bounds are best possible if A is strong Hall 15234 1 2 3 4 5 A 15234 1 5 2 3 4 chol (A T A) G  (A) + + + +

13. Column elimination tree Elimination tree of A T A (if no cancellation) Depth-first spanning tree of G  (A) Represents column dependencies in various factorizations 15234 1 5 4 2 3 A 15234 1 5 2 3 4 chol (A T A) T  (A) +

14. [Matlab 4.0, 1992] Left-looking column-by-column LU factorization Depth-first search to predict structure of each column Slow search limited speed BLAS-1 limited cache reuse... SuperLU: supernodal BLAS-2.5 LU UMFPACK: multifrontal BLAS-3 LU [Matlab 6.5, 2002] Ordering for nonsymmetric LU is still not well understood That story continues...

15. PRESENT

16. Support graph preconditioning Support graph preconditioning [http://www.cs.sandia.gov/~bahendr/support.html] Define a preconditioner B for matrix A Explicitly compute the factorization B = LU Choose nonzero structure of B to make factoring cheap (using combinatorial tools from direct methods) Prove bounds on condition number using both algebraic and combinatorial tools

17. Support graph preconditioner: example Support graph preconditioner: example [Vaidya] A is symmetric positive definite with negative off-diagonal nzs B is a maximum-weight spanning tree for A (with diagonal modified to preserve row sums) Preconditioning costs O(n) time per iteration Eigenvalue bounds from graph embedding: product of congestion and dilation Condition number at most n 2 independent of coefficients Many improvements exist G(A) G(B)

18. can improve congestion and dilation by adding a few strategically chosen edges to B cost of factor+solve is O(n 1.75 ), or O(n 1.2 ) if A is planar in experiments by Chen & Toledo, often better than drop-tolerance MIC for 2D problems, but not for 3D. G(A) G(B) Support graph preconditioner: example Support graph preconditioner: example [Vaidya]

19. Algebraic framework Algebraic framework [Gremban/Miller/Boman/Hendrickson] The support of B for A is σ(A, B) = min{ τ : x T (tB – A)x  0 for all x, all t  τ } In the SPD case, σ(A, B) = max{ λ : Ax = λ Bx} = λ max (A, B) Theorem 1: If A, B are SPD then κ (B -1 A) = σ(A, B) · σ(B, A) Theorem 2: If V·W=U, then σ(U·U T, V·V T )  ||W|| 2 2

20. Algebraic framework Algebraic framework [Gremban/Miller/Boman/Hendrickson] Lemma: If V·W=U, then σ(U·U T, V·V T )  ||W|| 2 2 Proof: take t  ||W|| 2 2 = λ max ( W·W T ) = max x  0 { x T W·W T x / x T x } then x T (tI - W·W T ) x  0 for all x letting x = V T y gives y T (tV·V T - U·U T ) y  0 for all y recall σ(A, B) = min{ τ : x T (tB – A)x  0 for all x, all t  τ } thus σ(U·U T, V·V T )  t

21. A B - a 2 - b 2 -a 2 -c 2 -b 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 +c 2 - c 2 - b 2 - c 2 b 2 +c 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 - b 2 b 2 [ ] a b - a c - b - c [ ] a b - a c - b U V =VV T =UU T [ ] 1 - c / a 1 c / b /b W = x σ(A, B)  ||W|| 2 2  ||W||  x ||W|| 1 = (max row sum) x (max col sum)  (max congestion) x (max dilation)

22. Open problems I Other subgraph constructions for better bounds on ||W|| 2 2 ? For example [Boman], ||W|| 2 2  ||W|| F 2 = sum(w ij 2 ) = sum of (weighted) dilations, and [Alon, Karp, Peleg, West] show there exists a spanning tree with average weighted dilation exp(O((log n loglog n) 1/2 )) = o(n  ); this gives condition number O(n 1+  ) and solution time O(n 1.5+  ), compared to Vaidya O(n 1.75 ) with augmented spanning tree Is there a construction that minimizes ||W|| 2 2 directly?

23. Open problems II Make spanning tree methods more effective in 3D? Vaidya: O(n 1.75 ) in general, O(n 1.2 ) in 2D Issue: 2D uses bounded excluded minors, not just separators Analyze a multilevel method in general? Extend to more general finite element matrices?

24. Link analysis of the world-wide web Web page = vertex Link = directed edge Link matrix: A ij = 1 if page i links to page j 1 2 3 4 7 6 5 1523467 1 5 2 3 4 6 7

25. Web graph: PageRank (Google) Web graph: PageRank (Google) [Brin, Page] Markov process: follow a random link most of the time; otherwise, go to any page at random. Importance = stationary distribution of Markov process. Transition matrix is p*A + (1-p)*ones(size(A)), scaled so each column sums to 1. Importance of page i is the i-th entry in the principal eigenvector of the transition matrix. But, the matrix is 2,000,000,000 by 2,000,000,000. An important page is one that many important pages point to.

26. Each page has hub score x i and authority score y i Repeat: y  A*x; x  A T *y; normalize; Converges to principal eigenvectors of AA T and A T A (left and right singular vectors of A) Web graph: Hubs and authorities Web graph: Hubs and authorities [Kleinberg] A good hub cites good authorities A good authority is cited by good hubs Hubs bikereviews.com phillybikeclub.org yahoo.com/cycling Authorities trekbikes.com shimano.com campagnolo.com

27. FUTURE

28. Combinatorial issues in numerical linear algebra Approximation theory for nonsymmetric LU ordering Preconditioning Complexity of matrix multiplication

29. Biology as an information science “The rate-limiting step of genomics is computational science.” - Eric Lander Sequence assembly, gene identification, alignment Large-scale data mining Protein modeling: discrete and continuous

30. Linear and sublinear algorithms for huge problems “Can we understand anything interesting about our data when we do not even have time to read all of it?” - Ronitt Rubinfeld

31. Fast Monte Carlo algorithms for finding low-rank approximations Fast Monte Carlo algorithms for finding low-rank approximations [Frieze, Kannan, Vempala] Describe a rank-k matrix B 0 that is within ε of the best rank-k approximation to A: ||A – B 0 || F  min B ||A - B|| F + ε ||A|| F Correct with probability at least 1 – δ Time polynomial in k, 1/ ε, log(1/δ); independent of size of A Idea: using a clever distribution, sample an O(k-by-k) submatrix of A and compute its SVD (Need to be able to sample A with the right distribution)

32. Approximating the MST weight in sublinear time Approximating the MST weight in sublinear time [Chazelle, Rubinfeld, Trevisan] Key subroutine: estimate number of connected components of a graph, in time depending on expected error but not on size of graph Idea: for each vertex v define f(v) = 1 / (size of component containing v) Then Σ v f(v) = number of connected components Estimate Σ v f(v) by breadth-first search from a few vertices

33. Approximate number of connected components for each of r randomly chosen vertices u 1... u r do: breadth-first search to distance 1 from u i repeat until b i is set if coin flip yields heads or BFS has reached at least w vertices then set b i = 0 and break repeat continue breadth-first search to double # vertices reached if BFS has reached all vertices in component containing u i then set b i = 2^(#flips) / (#vtxs reached) and break repeat end repeat end for return (n/r) * sum (b i )

34. Managing data and computation on the web Mapping, balancing, scheduling for parallel computing Data placement, compression, communication optimization Sophisticated cacheing (Akamai) Internet backplane (UTK) Active buffering, compressed migration (UIUC)

35. Modeling and distributed control Multiresolution modeling for nanomaterials and nanosystems Distributed control for systems on micro to global scales Imagining a molecular-scale crossbar switch [Heath et al., UCLA]

36. “Active surface” airjet paper mover “Active surface” airjet paper mover [Berlin, Biegelsen et al., PARC] sensors: 32,000 gray-level pixels in 25 linear arrays 576 valves (144 per direction) PC-hosted DSP control @ 1 kHz 12” x 12” board

37. A hard question How will combinatorial methods be used by people who don’t understand them in detail?

38. Matrix division in Matlab x = A \ b; Works for either full or sparse A Is A square? no => use QR to solve least squares problem Is A triangular or permuted triangular? yes => sparse triangular solve Is A symmetric with positive diagonal elements? yes => attempt Cholesky after symmetric minimum degree Otherwise => use LU on A(:, colamd(A))

39. Matching and depth-first search in Matlab dmperm: Dulmage-Mendelsohn decomposition Bipartite matching followed by strongly connected components Square, full rank A: [p, q, r] = dmperm(A); A(p,q) has nonzero diagonal and is in block upper triangular form also, strongly connected components of a directed graph also, connected components of an undirected graph Arbitrary A: [p, q, r, s] = dmperm(A); maximum-size matching in a bipartite graph minimum-size vertex cover in a bipartite graph decomposition into strong Hall blocks

40. A few themes Paths Locality Eigenvectors Huge data sets Multiple scales Usability

41. Morals Things are clearer if you look at them from two directions Combinatorial algorithms are pervasive in scientific computing and will become more so What are the implications for teaching? What are the implications for software development?

42. Thanks Patrick Amestoy, Erik Boman, Iain Duff, Mary Ann Branch Freeman, Bruce Hendrickson, Esmond Ng, Alex Pothen, Padma Raghavan, Ronitt Rubinfeld, Rob Schreiber, Sivan Toledo, Paul Van Dooren, Steve Vavasis, Santosh Vempala,...