1. Institute for Advanced Study, April 16 2012 Sushant Sachdeva Princeton University Joint work with Lorenzo Orecchia, Nisheeth K. Vishnoi Linear Time Graph Partitioning via Fast Simulation of Random walks

2. INTRODUCTION Theory of near-linear time algorithms

3. Why linear-time algorithms? Input graphs and data sets are massive. e.g. Web graph (10 11 nodes), Social networks (10 9 nodes) Super-linear algorithms are impractical. © 2004–2012 Michael K. Bergman. Need time algorithms for fundamental problems: e.g. M AX -M ATCHING, s-t M IN C UT /M AX F LOW, Balanced Graph Partitioning Approximate solutions maybe? (1- ² ) approx., time [Duan and Pettie] (1- ² ) approx., time [Duan and Pettie] (1- ² ) approx., time [Christiano et al.] (1- ² ) approx., time [Christiano et al.] Spectral approx., time [This talk] Spectral approx., time [This talk]

4. GRAPH PARTITIONING

5. Conductance Given a undirected, unweighted graph S Conductance of [Cheeger, Alon-Milman] [Spielman-Teng] allows us to approximate ¸ 2 (L) and find such a cut in time Assume G is d-regular Spectral approximation Conductance of G Fundamental quantity in Markov Chains, Riemannian Manifolds

6. Balanced Separator Given b, does G have a b-balanced cut of conductance < ° ? S G NP-Hard Applications to clustering, image segmentation, community detection, primitive for divide-and-conquer AlgorithmTechniqueDistinguishes ≤ ° and Running Time Recursive EigenvectorSpectral Use Cheeger to find a cut and remove it. Rinse and repeat. Can take time. Too slow. Use Cheeger to find a cut and remove it. Rinse and repeat. Can take time. Too slow.

7. Balanced Separator Given b, does G have a b-balanced cut of conductance < ° ? S G NP-Hard AlgorithmTechniqueDistinguishes ≤ ° and Running Time Recursive EigenvectorSpectral [Leighton-Rao]Flow [AK,OSVV,CKM + ] [Arora-Rao-Vazirani]SDP[AK,She,CKM + ] [Madry]

8. Balanced Separator Given b, does G have a b-balanced cut of conductance < ° ? S G NP-Hard AlgorithmTechniqueDistinguishes ≤ ° and Running Time Recursive EigenvectorSpectral [Spielman-Teng]Local Random walks [Andersen-Peres]Evolving Sets [Orecchia-Vishnoi]SDP

9. Balanced Separator Given b, does G have a b-balanced cut of conductance < ° ? S G NP-Hard AlgorithmTechniqueDistinguishes ≤ ° and Running Time Recursive EigenvectorSpectral [Spielman-Teng]Local Random walks [Andersen-Peres]Evolving Sets [Orecchia-Vishnoi]SDP [This talk]Simulating Random walks

10. Balanced Separator Given b, does G have a b-balanced cut of conductance < ° ? S G NP-Hard AlgorithmTechniqueDistinguish ≤ ° and Running Time Recursive EigenvectorSpectral [Leighton-Rao]Flow [AK,OSVV,CKM + ] [Arora-Rao-Vazirani]SDP[AK,She,CKM + ] [Madry] [This talk] Simulating random walks

11. Overview 1.Use continuous-time random walks to give an algorithm 2.Show how to simulate continuous-time random walks fast

12. Graph Cuts and Random Walks Intimate connection between Graph cuts and Random walks. [Alon-Milman][Mihail][Lovasz-Simonovits][Spielman-Teng] [Anderson-Chung-Lang] [Andersen-Peres] [Orecchia-Vishnoi][this talk]… [Mihail] Given the distribution of a random walk that has mixed, can find a cut S with in time. Usual Random Walks on a graph Page Rank Random walks Page Rank Random walks Evolving Sets Random Walks Continuous Time Random Walks

13. Bird’s eye view 1. Simulate random walks (one or several) 3. Else, modify the graph Regular walk / Page Rank walk / Evolving Sets / Continuous-time walks 2. Try to cut the graph using the random walks If there’s no low conductance cut, we get a certificate. If you find a balanced low conductance cut, we’re done. To remove the unbalanced low conductance cuts Remove the setAdd edges across the cut Using matchings / flows / stars Threshold cuts/ SDP rounding

14. Continuous Time Random Walks Thm. We can reduce Balanced Separator to simulating heat-kernel random walks with Thm. We can reduce Balanced Separator to simulating heat-kernel random walks with Given initial distribution, Random walk of length l ~ Poisson(t) Heat-Kernel random walk Why heat-kernel random walk? e -x is a nice function. Track progress using Golden Thompson inequality [AK] Don’t know such a result for other random walks Techniques from SDPs[OV], exponential updates aka Matrix Mult. Weights [AK] Previous results had a poly(1/ ° ) dependence

15. Algorithm We j An embedding in Random walk from vertex i Case 1: All vectors are small Long vectors are random walks that haven’t mixed We i All random walks have mixed well. No small cuts. Ensures mixing across cuts with conductance more than °

16. Algorithm Case 2: Walks have not mixed We j We i Random Projection + Threshold Cut “Cheeger Rounding” If we find a balanced cut of small conductance, we’re done. ELSE

17. Algorithm Case 2b: Walks have not mixed (and we didn’t find a balanced cut) We j We i We can find a ball cut S S S is the union of “all unbalanced low conductance cuts”

18. Algorithm Case 2b: Walks have not mixed (and we didn’t find a balanced cut) We j We i We can find a ball cut S S Add edges from all vertices outside the ball to all inside Soft removal of unbalanced cuts O(log n) rounds For efficiency, we use JL lemma and work with O(log n) dimensional embeddings. [AK] For efficiency, we use JL lemma and work with O(log n) dimensional embeddings. [AK] Goal: Given L, ¿,v. Compute Goal: Given L, ¿,v. Approximate

19. SIMULATING RANDOM WALKS

20. Approximating Heat-Kernel Goal: Given ¿, L, v, ±, find u s.t. Goal: Given ¿, L, v, ±, find u s.t. Taylor Series Approx.? Need Too Slow Heat-Kernel random walk required by algorithm Small error suffices for the algorithm Used in previous algorithms

21. Approximating Heat-Kernel Better polynomials? YES!! [Thm] There exists a polynomial p of degree such that [Thm] There exists a polynomial p of degree such that It suffices to approximate e -x on [Thm] Given G that has a balanced cut of conductance, we can find one of conductance in time Better guarantee, same running time as [Andersen-Peres] Still not near-linear! Goal: Given ¿, L, v, ±, find u s.t. Goal: Given ¿, L, v, ±, find u s.t.

22. Approximating Heat-Kernel [Thm] Any polynomial such that requires degree [Thm] Any polynomial such that requires degree Even better polynomials? Not Really! Simple proof using Markov’s theorem Simple proof using Markov’s theorem Goal: Given ¿, L, v, ±, find u s.t. Goal: Given ¿, L, v, ±, find u s.t.

23. Beyond Polynomials Rational Functions! [Saff-Schönhage-Varga] There is a degree k polynomial p k such that Infinite Interval! Geometric Decay! Need degree Geometric Decay! Need degree Assume: 1. We knew p k explicitly 2. ST computation was exact Assume: 1. We knew p k explicitly 2. ST computation was exact We’re DONE! Two Issues: 1. [SSV] result is existential 2. ST computation is approximate Two Issues: 1. [SSV] result is existential 2. ST computation is approximate Approximating up to error ± Approximating up to error ± Computing vectors Can use [Spielman-Teng]! [Thm] Given ¿, L, v, ±, we can compute u such that in time [Thm] Given ¿, L, v, ±, we can compute u such that in time [Spielman-Teng] (Informal) We can approximate (cI+L) -1 y in time. [Spielman-Teng] (Informal) We can approximate (cI+L) -1 y in time.

24. LANCZOS’ METHOD “But we don’t know the polynomial.”

25. Lanczos’ Method Let Define f so that Goal: Approximate f(B)v as well as any degree k poly. Lanczos Method Existence of a good approximate poly suffices Small degree implies efficiency From real approximation to matrices

26. Lanczos’ Method Goal: Approximate f(B)v as well as any degree k poly. Observe p – any degree k polynomial Krylov Subspace · of order k Let be an orthonormal basis for · Let T be the operator B restricted to · (k+1) x (k+1) matrix For any degree k polynomial p, For any degree k polynomial p, Let Projection on · Our guess: T is a much smaller matrix!

27. Error Guarantee Goal: Bound the norm of the error Let the error be Equal! Since T is B restricted to ·, Spectrum(T) is bounded by Spectrum(B) Since T is B restricted to ·, Spectrum(T) is bounded by Spectrum(B) * error of best degree k polynomial approximation * error of best degree k polynomial approximation Cost = k mult. by B + Construct basis + Diagonalize T for our setting Cost = k mult. by B + Construct basis + Diagonalize T for our setting Suppose p(x) is a good degree k approximation to f Bounded by r on Spectrum(B) Bounded by r on Spectrum(T) Don’t even need to know the polynomial!

28. HANDLING APPROXIMATE COMPUTATION What about the error?

29. We bound the error Why Lanczos is insufficient Multiplication with is only approximate Multiplication with is only approximate Subspace · is approximate. & Operator T is approximate Subspace · is approximate. & Operator T is approximate T is not even symmetric! We lose nice spectral properties [ST] computation is only approximate We symmetrize. Compute approximation with We symmetrize. Compute approximation with We needed to compute an orthonormal basis for

30. Sketch of Error Analysis Bound Polynomial part Error part Rational approx. result Bound Bound sum of coefficients of p k Bound spectrum of Bound spectrum of Error Analysis + Lanczos method = Fast random walks Fast random walks + Algorithm = Fast Balanced Separator

31. Conclusions Reduced balanced separator to simulating heat-kernel random walks Approximated the heat kernel random walk in time. Can be used as primitives for designing near- linear time algorithms.

32. Open Questions Other applications of fast matrix exponentiation Linear-time Graph decomposition? Linear time algorithms for small set expansion? Linear time algorithm for approximation for Balanced Separator

33. Thank you!

34. An Important Tool [Christiano et al.] s-t M AX F LOW s-t M IN C UT s-t M AX F LOW s-t M IN C UT Electrical Flows Solve Linear Systems Lx=y L is a graph Laplacian Solving Laplacian Linear Systems Direct Methods Gaussian Elimination / Cholesky Decomposition Gaussian Elimination / Cholesky Decomposition Iterative methods Too slow. O(n 3 ) time Hopeless Too slow. O(n 3 ) time Hopeless Conjugate Gradient Approximate method. time Approximate method. time [Spielman-Teng] (Informal) We can approximate L -1 y in time. [Spielman-Teng] (Informal) We can approximate L -1 y in time. Other Applications – Approximating second eigenvector [Spielman-Teng], Cover times [Ding-Lee-Peres], Generating random spanning trees[Madry-Kelner], Graph Sparsification [Spielman-Srivastava] Balanced graph partitioning, Simulating continuous-time random walks [This talk] Balanced graph partitioning, Simulating continuous-time random walks [This talk]

35. Beyond Polynomials Rational Functions! [Saff-Schönhage-Varga] There is a degree k polynomial p k such that Infinite Interval! Geometric Decay! Small required degree. Geometric Decay! Small required degree. L 1 bound on [0, 1 ) L 1 bound on [-1,1] L 2 bound on [-1,1] Change IntervalWrite function as an integralCauchy Schwarz Assume: 1. We knew p k explicitly 2. ST Solver was exact Assume: 1. We knew p k explicitly 2. ST Solver was exact We’re DONE! Two Issues: 1. We don’t know p k explicitly 2. ST Solver is approximate Two Issues: 1. We don’t know p k explicitly 2. ST Solver is approximate

36. Faster Simulation Polynomials don’t suffice, then, how do we simulate the random walk fast?

37. Lanczos Method Goal: Given symmetric B, v, k, and function f; approximate f(B)v as well as any degree k poly Goal: Given symmetric B, v, k, and function f; approximate f(B)v as well as any degree k poly Observe p – any degree k polynomial Krylov Subspace · of order k Let be an orthonormal basis for · k-Identity Projection on · Let For any degree k polynomial p, For any degree k polynomial p, Our guess: T is a much smaller matrix!

38. Error Guarantee Goal: Bound the norm of the error Write Any degree k polynomial Equal! Since and the columns of V are orthnormal, The columns of V are orthnormal True for any p!! As good as the best degree k polynomial! Cost = k mult. by B + Construct basis + Diagonalize T if B is a Laplacian Cost = k mult. by B + Construct basis + Diagonalize T if B is a Laplacian

39. Using Lanczos for our Algorithm Goal: Given A, wanted to approximate as Goal: Given A, wanted to approximate as Use Lanczos with suffices. Still need to address the issue of approximate multiplication with