1. Sparse Approximations
Nick Harvey University of British Columbia
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAA

2. Approximating Dense Objects by Sparse Objects
Floor joists
Wood Joists
Engineered Joists

3. Approximating Dense Objects by Sparse Objects
Bridges
Masonry Arch
Truss Arch

4. Approximating Dense Objects by Sparse Objects
Bones
Human Femur
Robin Bone

5. Mathematically
Can an object with many pieces be approximately represented by fewer pieces?
Independent random sampling usually does well
Theme of this talk: When can we beat random sampling? 6 -1 4 5 7 6 -1 5 -3 2 8 1
Dense Matrix
Sparse Matrix
Dense Graph
Sparse Graph

6. Talk Outline Vignette #1: Discrepancy theory
Vignette #2: Singular values and eigenvalues
Vignette #3: Graphs
Theorem on “Spectrally Thin Trees”

7. Discrepancy
Given vectors v1,…,vn2Rd with kvikp bounded. Want y2{-1,1}n with ki yivikq small.
Eg1: If kvik1·1 then E ki yi vik1 ·
Eg2: If kvik1·1 then 9y s.t. ki yi vik1 ·
Spencer ‘85: Partial Coloring + Entropy Method
Gluskin ‘89: Sidak’s Lemma
Giannopoulos ‘97: Partial Coloring + Sidak
Bansal ‘10: Brownian Motion + Semidefinite Program
Bansal-Spencer ‘11: Brownian Motion + Potential function
Lovett-Meka ‘12: Brownian Motion
Non-algorithmic
Algorithmic

8. Discrepancy
Given vectors v1,…,vn2Rd with kvikp bounded. Want y2{-1,1}n with ki yivikq small.
Eg1: If kvik1·1 then E ki yi vik1 ·
Eg2: If kvik1·1 then 9y s.t. ki yi vik1 ·
Eg3: If kvik1·¯, kvik1·±, and ki vik1·1, then 9y with ki yi vik1 ·
Harvey ’13: Using Lovasz Local Lemma.
Question: Can log(±/¯2) factor be improved?

9. Talk Outline Vignette #1: Discrepancy theory
Vignette #2: Singular values and eigenvalues
Vignette #3: Graphs
Theorem on “Spectrally Thin Trees”

10. Partitioning sums of rank-1 matrices
Let v1,…,vn2Rd satisfy i viviT=I and kvik2·±. Want y2{-1,1}n with ki yiviviTk2 small.
Random sampling: E ki yiviviTk2 ·
Rudelson ’96: Proofs using majorizing measures, then nc-Khintchine
Marcus-Spielman-Srivastava ’13: 9y2{-1,1}n with ki yiviviTk2 · 2

11. Partitioning sums of matrices
Given dxd symmetric matrices M1,…,Mn2Rd with i Mi=I and kMik2·±. Want y2{-1,1}n with ki yiMik2 small.
Random sampling: E ki yiMik2 ·
Also follows from nc-Khintchine.
Ahlswede-Winter ’02: Using matrix moment generating function.
Tropp ‘12: Using matrix cumulant generating function.

12. Partitioning sums of matrices
Given dxd symmetric matrices M1,…,Mn2Rd with i Mi=I and kMik2·±. Want y2{-1,1}n with ki yiMik2 small.
Random sampling: E ki yiMik2 ·
Question: 9y2{-1,1}n with ki yiMik2 · ?
Conjecture: Suppose i Mi=I and kMikSch-1·±. 9y2{-1,1}n with ki yiMik2 · ?
MSS ’13: Rank-one case is true
Harvey ’13: Diagonal case is true (ignoring log(¢) factor)
False!

13. Partitioning sums of matrices
Given dxd symmetric matrices M1,…,Mn2Rd with i Mi=I and kMik2·±. Want y2{-1,1}n with ki yiMik2 small.
Random sampling: E ki yiMik2 ·
Question: Suppose only that kMik2·1. 9y2{-1,1}n with ki yiMik2 · ?
Spencer/Gluskin: Diagonal case is true

14. Column-subset selection
Given vectors v1,…,vn2Rd with kvik2=1.
Let st.rank=n/ki viviTk2. Let
9y2{0,1}n s.t. i yi=k and (1-²)2 · ¸k( i yiviviT ).
Spielman-Srivastava ’09: Potential function argument
Youssef ’12: Let y2{0,1}n s.t. i yi=k, (1-²)2 · ¸k( i yiviviT ) and ¸1( i yiviviT ) · (1+²)2.

15. Column-subset selection up to the stable rank
Given vectors v1,…,vn2Rd with kvik2=1. Let st.rank=n/ki viviTk2. Let For y2{0,1}n s.t. i yi=k, can we control ¸k( i yiviviT ) and ¸1( i yiviviT ) ?
¸k can be very small, say O(1/d).
Rudelson’s theorem: can get ¸1 · O(log d) and ¸k>0.
Harvey-Olver ’13: ¸1 · O(log d / log log d) and ¸k>0.
MSS ‘13: If i viviT =I, can get ¸1 · O(1) and ¸k>0.

16. Talk Outline Vignette #1: Discrepancy theory
Vignette #2: Singular values and eigenvalues
Vignette #3: Graphs
Theorem on “Spectrally Thin Trees”

17. weighted degree of node c
Graph Laplacian 5 10
Graph with weights u: a c d 2 1 b a b c d
Laplacian Matrix: a 7 -2 -5 3 -1 16
-10 10
negative of u(ac) b
Lu = D-A = c
weighted degree of node c d
Effective Resistance from s to t: voltage difference when each edge e is a (1/ue)-ohm resistor and a 1-amp current source placed between s and t = (es-et)T Luy (es-et) Effective Conductance: cst = 1 / (effective resistance from s to t)

18. Spectral approximation of graphs
Edge weights u
Edge weights w 5 -1 4 6 7 6 -1 -5 5 -3 2 8 -8 1 10
Lu =
Lw =
®-spectral sparsifier: Lu ¹ Lw ¹ ®¢Lu

19. Ramanujan Graphs Suppose Lu is complete graph on n vertices (ue=1 8e).
Lubotzky-Phillips-Sarnak ’86: For infinitely many d and n, 9w2{0,1}E such that e we=dn/ (actually Lw is d-regular) and
MSS ‘13: Holds for all d¸3, and all n=c¢2k.
Friedman ‘04: If Lw is a random d-regular graph, then 8²>0
with high probability.

20. Arbitrary graphs
Spielman-Srivastava ’08: For any graph Lu with n=|V|, 9w2RE such that |support(w)| = O(n log(n)/²2) and
Proof: Follows from Rudelson’s theorem
MSS ’13: For any graph Lu with n=|V|, 9w2RE such that we 2 £(²2) ¢ N ¢ (effective conductance of e) |support(w)| = O(n/²2) and

21. Spectrally-thin trees
Question: Let G be an unweighted graph with n vertices. Let C = mine (effective conductance of edge e). Want a subtree T of G with
Equivalent to
Goddyn’s Conjecture ‘85: There is a subtree T with
Relates to conjectures of Tutte (‘54) on nowhere-zero flows, and to approximations of the traveling salesman problem.

22. Spectrally-thin trees
Question: Let G be an unweighted graph with n vertices. Let C = mine (effective conductance of edge e). Want a subtree T of G with
Rudelson’s theorem: Easily gives ® = O(log n).
Harvey-Olver ‘13: ® = O(log n / log log n). Moreover, there is an efficient algorithm to find such a tree.
MSS ’13: ® = O(1), but not algorithmic.

23. Talk Outline Vignette #1: Discrepancy theory
Vignette #2: Singular values and eigenvalues
Vignette #3: Graphs
Theorem on “Spectrally Thin Trees”

24. Spectrally Thin Trees Proof overview:
Given an (unweighted) graph G with eff. conductances ¸ C.
Can find an unweighted tree T with
Proof overview:
Show independent sampling gives spectral thinness, but not a tree.
► Sample every edge e independently with prob. xe=1/ce
Show dependent sampling gives a tree, and spectral thinness still works.

25. Matrix Concentration
Theorem: [Tropp ‘12] Let Y1,…,Ym be independent, PSD matrices of size nxn. Let Y=i Yi and Z=E [ Y ]. Suppose Yi ¹ R¢Z a.s. Then

26. Independent sampling Let Suppose Mi ¹ Z.
Define sampling probabilities xe = 1/ce. It is known that e xe = n–1.
Claim: Independent sampling gives T µ E with E [|T|]=n–1 and
Theorem [Tropp ‘12]: Let M1,…,Mm be nxn PSD matrices.
Let D(x) be a product distribution on {0,1}m with marginals x.
Let Suppose Mi ¹ Z.
Then
Define Me = ce¢Le. Then Z = LG and Me ¹ Z holds.
Setting ®=6 log n / log log n, we get whp.
But T is not a tree!
Yi = Xi Mi
Y = \sum_i Yi
Z = E[Y] = E[\sum_i Yi] = \sum_i E[Xi Mi] = \sum_i xi Mi.
Assume that Yi <= R*Z as, which is equivalent to Mi <= R*Z a.s.
It turns out that R = 1.
Laplacian of the single edge e
Properties of conductances used

27. Spectrally Thin Trees Proof overview:
Given an (unweighted) graph G with eff. conductances ¸ C.
Can find an unweighted tree T with
Proof overview:
Show independent sampling gives spectral thinness, but not a tree.
► Sample every edge e independently with prob. xe=1/ce
Show dependent sampling gives a tree, and spectral thinness still works.
► Run pipage rounding to get tree T with Pr[ e2T ] = xe = 1/ce

28. Pipage rounding
[Ageev-Svirideno ‘04, Srinivasan ‘01, Calinescu et al. ‘07, Chekuri et al. ‘09]
Let P be any matroid polytope. E.g., convex hull of characteristic vectors of spanning trees.
Given fractional x
Find coordinates a and b s.t. line z  x + z ( ea – eb ) stays in current face
Find two points where line leaves P
Randomly choose one of those points s.t. expectation is x
Repeat until x = ÂT is integral
x is a martingale: expectation of final ÂT is original fractional x.
ÂT1
ÂT6
ÂT2 x
ÂT3
ÂT5
ÂT4

29. Pipage rounding and concavity
Say f : Rm ! R is concave under swaps if z ! f( x + z(ea-eb) ) is concave 8x2P, 8a, b2[m].
Let X0 be initial point and ÂT be final point visited by pipage rounding.
Claim: If f concave under swaps then E[f(ÂT)] · f(X0). [Jensen]
Let E µ {0,1}m be an event.
Let g : [0,1]m ! R be a pessimistic estimator for E, i.e.,
Claim: Suppose g is concave under swaps. Then Pr[ ÂT 2 E ] · g(X0).

30. Chernoff Bound Chernoff Bound: Fix any w, x 2 [0,1]m and let ¹ = wTx.
Define Then,
Claim: gt,µ is concave under swaps [Elementary calculus]
Let X0 be initial point and ÂT be final point visited by pipage rounding.
Let ¹ = wTX0. Then
Bound achieved by independent sampling also achieved by pipage rounding

31. Matrix Pessimistic Estimators
Theorem [Tropp ‘12]: Let M1,…,Mm be nxn PSD matrices.
Let D(x) be a product distribution on {0,1}m with marginals x.
Let Suppose Mi ¹ Z.
Let
Then
and
Pessimistic estimator
Main Theorem: gt,µ is concave under swaps.
Bound achieved by independent sampling also achieved by pipage rounding

32. Spectrally Thin Trees Proof overview:
Given an (unweighted) graph G with eff. conductances ¸ C.
Can find an unweighted tree T with
Proof overview:
Show independent sampling gives spectral thinness, but not a tree.
► Sample every edge e independently with prob. xe=1/ce
Show dependent sampling gives a tree, and spectral thinness still works.
► Run pipage rounding to get tree T with Pr[ e2T ] = xe = 1/ce

33. Matrix Analysis
Matrix concentration inequalities are usually proven via sophisticated inequalities in matrix analysis Rudelson: non-commutative Khinchine inequality Ahlswede-Winter: Golden-Thompson inequality if A, B symmetric, then tr(eA+B) · tr(eA eB). Tropp: Lieb’s concavity inequality [1973] if A, B Hermitian and C is PD, then z ! tr exp( A + log(C+zB) ) is concave. Key technical result: new variant of Lieb’s theorem if A Hermitian, B1, B2 are PSD, and C1, C2 are PD, then z ! tr exp( A + log(C1+zB1) + log(C2–zB2) ) is concave.

34. Questions
Can Spencer/Gluskin theorem be extended to matrices? Can MSS’13 be made algorithmic? Can MSS’13 be extended to large-rank matrices? O(1)-spectrally thin trees exist. Can one be found algorithmically? Are O(1)-spectrally thin trees helpful for Goddyn’s conjecture?