1. Algorithm Frameworks Using Adaptive Sampling Richard Peng Georgia Tech

2. OUTLINE Sampling and its algorithmic incorporation Adaptive sampling for approximating maxflow Adaptive sampling for solving linear systems

3. RANDOM SAMPLING Pick a small subset of a collection of many objects Goal: Estimate quantities Reduce sizes Speed up algorithms

4. ALGORITHMIC USE OF SAMPLING Framework: Compute on the sample Bring answer back to original Original Compute on sample Examples: Quick sort/select Geometric cuttings Nystrom method

5. THIS TALK Adaptive sampling based algorithms for finding combinatorial and scientific flows Current best, O(mlog c n) time guarantees for: Approximate maxflow / balanced cuts Computation of random walks / eigenvectors Measure runtime with big-O, address worst case instances, including ill-conditioned ones

6. ADAPTIVE SAMPLING Iterative schemes: chain of samples, each constructed from the previous Preserve `just enough’ for answer on the sample to be useful Problem 1 Problem 2 Problem d …

7. PRESERVING GRAPH STRUCTURES This talk: undirected graphs n vertices m < n 2 edges Is n 2 edges (dense) sometimes necessary? For connectivity: < n edges always ok

8. PRESERVING MORE [BK `96]: for ANY G, can get H with O(nlogn) edges s.t. G ≈ H on all cuts Andras Benczur David Karger How: keep edge e with probability p e, rescale if kept to maintain expectation

9. HOW TO PICK PROBABILITIES Widely used: uniform sampling Works well when data is uniform e.g. complete graph Problem: long path, removing any edge changes connectivity (can also have both in one graph) Fix: pick probaiblities adaptively baesd on graph

10. THE `RIGHT’ PROBABILITIES Path + clique: 1 1/n [RV `07], [Tropp `12], [SS `08]: suffices to have p e ≥ O( logn) × w e × effective resistance Dan Spielman Nikhil Srivastava Gives spectral sparsifiers, which suffices for all the results in this talk, e.g. preserves cuts

11. SAMPLING MORE [Talagrand `90] “Embedding subspaces of L 1 into L N 1 ” : non-linear, matrix, analog, O(dlogd) objects for d dimensions Mathematical view of probabilities: 2-norm: leverage scores 1-norm: [CP `15]: Lewis weights ║x║1║x║1 ║x║2║x║2 Michel Talagrand

12. COMPUTING SAMPLING PROBABILITIES [ST `04][OV `11]: spectral partitioning [SS`08][KLP `12]: projections + solves [Koutis `14]: spanners / low diameter partitions [LMP `13][CLMMPS`15]: self-reduction / recursion [DMMW`13][CW`13]: sketches [BSS`09][LS `15]: potential functions For edge with Laplacian L e w e × r e = trace( L + L e ) +: pseudo-inverse

13. OUTLINE Sampling and its algorithmic incorporation Adaptive sampling for approximating maxflow Adaptive sampling for solving linear systems

14. s Undirected graphs PROBLEM t Maximum number of disjoint s-t paths Applications: Routing Scheduling Dual: separate s and t by removing fewest edges Applications: Partitioning Clustering n vertices, m edges Assume capacities = poly(n) Goal: 1± ε approx in O(mlog c nε -2 ) time

15. May also ‘fix’ earlier flow via residual graph Faster: [EK `73]: fewer paths: O(m 2 n) [Dinic `73][HK `75] [GN `80][ST `83]: shorter / simpler paths, O(nmlogn) ‘PROTOTYPE’ FLOW ALGORITHM (FORD-FULKERSON `56) While exists s-t path route it adjust graph s t s t D. R. Fulkerson L. R. Ford Jr.

16. EXTREME INSTANCES Highly connected, need global steps Long paths / tree, need many steps But must handle both simultaneously gradient steps MCMC Power method min-degree dynamic trees DFS Each easy on their own

17. John Hopcroft LIMIT OF THIS APPROACH? s t s t [HK `75]: if we can route f units of flow in a unit-capacity graph, the shortest augmenting path has length at most m / f If we can find (approximate) shortest paths dynamically: Σ f=1 m (m / f) = O(mlogn) Richard Karp

18. 1980: dynamic trees 1970s: blocking flows 1986: dual algorithms 1999: scaling 2010: linear systems 1956: augmenting paths Augmenting paths [FF `55] Notion of polynomial time [Edmonds `65] Blocking flows [EK`75, Dinic `73] Dynamic tree data structures [GN `80] Dual/scaling algorithms [Gabow `83, GT `88, GR `99] Faster solvers for graph Laplacians [Vaidya `89] WORK RELATED TO FLOWS LED TO:

19. AVERAGE TOGETHER FLOWS NUMERICALLY Numerical notions of `approximate’ shortest paths, interact well with Hierarchical decompositions, Divide-and-conquer on graphs s s t t s t

20. Jon Kelner NUMERICAL MAXFLOW ALGORITHMS [Sherman `13] [KLOS `14]: given operator that α-approximates maxflow for ANY demand d, can compute (1 + ε)-approx maxflow in O(α 2 lognε -2 ) calls [Madry `10] [KLOS `13]: build this operator with α = O(m θ ) in O(m 1+θ ) time Jonah Sherman Alexander Madry Aaron Sidford Yin-Tat Lee Lorenzo Orecchia Combining gives O(m 1+θ ), will show next: O(mlog c n)

21. -2 3 OPERATOR? Tree: unique s-t path, maxflow = demand / bottleneck edge s t 1 -2 Multiple (exact) demands: flows along each edge determined via linear mapping, O(n) time

22. TREE FOR ANY GRAPH? [RST `14]: can find such a tree by solving maxflows of total size O(mlog c n) [Racke `01]: ANY undirected graph has a tree that’s an O(log c n) approximator Harald Racke Hanjo Taubig Chintan Shah

23. USING THESE DECOMPOSITIONS? Approximator Maxflow [RST `14] operator with α=O(log c n) via maxflows of total size O(mlog c n) Maxflow Approximator O(mlog c nε -2 ) time? [Sherman `13] [KLOS `14]: given operator that α-approximates maxflow for ANY demand d, can compute (1 +ε)-approx maxflow in O(α 2 lognε -2 ) calls

24. RESOLUTION: RECURSION Create smaller approximation Build cut/flow approximator on it recursively Use (fixed size) approximator for the original ` Adaptive sampling acts driver that controls progress of recursion

25. SIZE REDUCTION Ultra-sparsifier: for any k, can find H ≈ k G that’s tree + O(mlog c n/k) edges ` ` e.g. [Koutis-Miller-P `10]: pick good tree, sample off-tree edges by their ‘stretch’ Reducible to O(mlog c n/k) vertices/edges Yiannis Koutis Gary Miller

26. Use: maxflow from O(α 2 lognε -2 ) calls Construction: α=O(log c n) via maxflows on graphs of total size O(mlog c n) [P `16]: INCORPORATING REDUCTIONS T(m) = T(mlog 2c n/k) + O(mk 2 log 2c+1 n) Set k  O(log 2c n): T(m) = T(m/2) + O(mlog 4c+1 n) = O(mlog 4c+1 n) Construct on ultra-sparsifier with approximation factor k α=O(klog 2c n) for original graph New size = O(mlog c n/k) Maxflow

27. OUTLINE Sampling and its algorithmic incorporation Adaptive sampling for approximating maxflow Adaptive sampling for solving linear systems

28. GRAPH LAPLACIANS 1 1 2 -1 -1 -1 1 0 -1 0 1 Matrices that correspond to undirected graphs Entries  vertices, n by n matrices Non-zeros  edges, O(m) nonzeros Problem: given graph Laplacian L, vector b, find x s.t. Lx = b

29. THE LAPLACIAN PARADIGM Directly related : Elliptic systems Few iterations : Eigenvectors, Heat kernels Many iterations / modify algorithm Graph problems Image processing Dan Spielman Shanghua Teng

30. USE MATLAB? TRILINOS? LAPACK? Sequence of (adpatively) generated linear systems: https://github.com/sachdevasushant/Isotonic/blob/master/README.mdhttps://github.com/sachdevasushant/Isotonic/blob/master/README.md : …we suggest rerunning the program a few times and/or using a different solver. An alternate solver based on incomplete Cholesky factorization is provided… Optimization Problem Linear System Solver Kevin Deweese

31. SIMPLIFICATION Adjust/rescale so diagonal = I Add to diagonal to make full rank L = I – A A: Random walk

32. LOWER BOUND FOR ITERATIVE METHODS Graph theoretic interpretation: each term  1 step walk A diameter b bAbA2bA2b Need Ω(diameter) steps Division with multiplication: L -1 = ( I – A ) -1 = I + A + A 2 + A 3 +…

33. ( I – A ) -1 = I + A + A 2 + A 3 + … = ( I + A ) ( I + A 2 ) ( I + A 4 )… REPEATED SQUARING Dense matrix! A 16 = (((( A 2 ) 2 ) 2 ) 2, 4 operations O(logn) terms ok Similar to multi-level methods

34. DENSE INTERMEDIATE OBJECTS Matrix powers Matrix inverse Transitive closures LU factorizations Cost-prohibitive to store / find But can access a sparse version

35. GRAPH THEORETIC VIEW A : step of random walk A 2 : 2 step random walk Still a graph! [PS `14]: can directly access a sparsifier in O(mlog c n) time

36. HIGHER POWERS A : random walk A k : k step random walk [CCLPT `15: can also compute this sparsifier in nearly-linear time Shanghua Teng Yan Liu Yu Cheng Dehua Cheng Sparse graph close to A k

37. SPARSIFIED SQUARING I - A 1 ≈ ε I – A 2 I – A 2 ≈ ε I – A 1 2 … I – A i ≈ ε I – A i-1 2 I - A d ≈ I I - A 0 I - A d ≈ I Convergence: (approximately) same as repeated squaring: d = O(log(mixing time)) suffices ≈: spectral/condition number, implies cut

38. [PS `14] CHAIN  SOLVER x = Solve( I, A 0, … A d, b) 1.For i =1 to d, b i  ( I + A i ) b i-1. 2.x d  b d. 3.For i = d - 1 downto 0, x i  ½[b i +( I + A i )x i+1 ]. Runtime: O(mlog c nlog 3 (mixing time))

39. SPARSE BLOCK CHOLESKY [KLPRS`16]: Repeatedly eliminate some variables Sparsify intermediate matrices O(mlogn) time, extends to connection Laplacians, which can be viewed as having complex weights Rasmus Kyng Sushant Sachdeva

40. EVEN MORE ADAPTIVE [KS `16] Per-entry pivoting, almost identical to incomplete LU / ichol Running time bound: O(mlog 3 n), OPEN: improve this

41. OPEN QUESTIONS Nearly-linear time algorithms for: Wider classes of linear systems Directed maximum flow Intermediate questions: Squaring based flow algorithms / oblivious routing schemes. What can we preserve while sparsifying directed graphs?

42. THANK YOU!