Matrix sparsification and the sparse null space problem Lee-Ad GottliebWeizmann Institute Tyler NeylonBynomial Inc. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

Matrix sparsification and the sparse null space problem 2 Matrix sparsification Problem definition: Given a matrix, make it as sparse as possible (minimize number of non-zeros), using elementary row reductions  we want lots of 0’s Could try Gaussian elimination… but can we do better? Applications:  Computational speed-up for many fundamental matrix operations  Machine learning [SS-00]  Discovery of cycle bases of graphs [KMMP-04]

Matrix sparsification and the sparse null space problem 3 Matrix sparsification What’s known about matrix sparsification?  Precious little… mostly work by McCormick and coauthors It’s NP-hard [M-83] Known results  Heuristic [CM-02]  Algorithm under limiting condition: [HM-84] gave an approximation algorithm for matrices that satisfy the Haar condition

Matrix sparsification and the sparse null space problem 4 Sparse null space problem First recall the definition of null space:  The null space of A is the set of all nonzero vectors b for which Ab=0  A null matrix for A spans the null space of A. Finding such a matrix is a basic function in linear algebra. Problem definition: Given a matrix A, find an optimally sparse matrix N that is a full null matrix for A  N is full rank  Columns of N span the null space of A  N is sparse Applications  Helps solve Linear Equality Problems [CP-86] (optimization via gradient descent, dual variable method for Navier-Stokes, quadratic programming)  Efficient solution to the force method for structural analysis [GH-87]  Finds correlations between time series, such as financial stocks [ZZNS-05]

Matrix sparsification and the sparse null space problem 5 Sparse null space problem What’s known about sparse null space?  Precious little… First considered in [P-84], it’s known to be NP-hard [CP-86] Known results: Heuristics [BHKLPW-85, CP-86, GH-87]

Matrix sparsification and the sparse null space problem 6 Two matrix problems It’s not difficult to see that matrix sparsification and sparse null space are equivalent  Let B be a full null matrix for A. The following statements are equivalent: N = BX for some invertible matrix X N is a full null matrix for X Surprisingly then, these two lines of work have proceeded independently

Matrix sparsification and the sparse null space problem 7 Our contribution Two matrix problems  Have been around since the 80’s  Have many applications  Are equivalent – from now on, we’ll refer only to matrix sparsification Can we prove something concrete about matrix sparsification?  Hardness of approximation?  Approximation algorithms? We can do both…  Hardness of approximation As hard as label cover (quite hard to approximate) Hard to approximate within factor 2 log.5-o(1) n of optimal (with some caveats…)  Approximation algorithms For the hard problem Under limiting assumptions

Matrix sparsification and the sparse null space problem 8 Min unsatisfy As a first step towards proving hardness of approximation, we’ll show that matrix sparsification is closely related to the min unsatisfy problem introduced in [ABSZ-97] Problem definition: Given a linear system Ax=b, provide a vector x that minimizes the number of equations not satisfied What’s known about min unsatisfy  As hard to approximate as label cover [ABSZ-97]  Under Q, hard within factor 2 log.5-o(1) n of optimal under the assumption that NP does not admit a quasi-polynomial time solution.  Randomized Θ(m/log m) approximation algorithm (m is number of rows) [BK-01] x1x1 x2x x =

Matrix sparsification and the sparse null space problem 9 Hardness of matrix sparsification We’ll give a reduction from min unsatisfy to matrix sparsification, which will prove hardness of approximation for matrix sparsification. Preliminary note: There exist matrices that are unsparsifiable. and these can be construction in poly time.  M = (I X), where I is the identity matrix and X contains no 0 entries.  The identity portion can always be achieved via Gaussian elimination

Matrix sparsification and the sparse null space problem 10 Hardness of matrix sparsification Proof outline  Let (A,y) be an instance of min unsatisfy  We’ll create a matrix M with a few copies of A, but many of copies of y  Minimizing the number of non-zero entries in M reduces to finding a sparse linear combination of y with some vectors of A That is, solving the instance of min unsatisfy. Construction: Let (I q X) be an unsparsifiable matrix, and Ø be the Kronecker product We chose q=n 2 I q Ø yX Ø y 0I q Ø A

Matrix sparsification and the sparse null space problem 11 Approximation algorithm Our first result: We conclude that matrix sparsification is as hard as min unsatisfy, which itself is as hard as label cover.  Matrix sparsification is hard to approximate within factor 2 log.5-o(1) n of optimal So what can be done for matrix sparsification? We will further show that a solution to min unsatisfy implies a similar solution for matrix sparsification.  Hence, the randomized Θ(m/log m) approximation algorithm for min unsatisfy [BK-01] carries over to matrix sparsification.  More importantly, we will also show how to extend a large number of heuristics and algorithms under limiting assumptions to apply to min unsatisfy, and therefore to matrix sparsification. In particular, we’ll show that the well-known l 1 minimization heuristic applies to matrix sparsification.

Matrix sparsification and the sparse null space problem 12 Another look at min unsatisfy Consider the exact dictionary representation (EDR) problem, the major problem in sparse approximation theory.  Problem definition: Given a matrix D of dictionary vectors and a target vector s  Find the smallest subset D’ such that a linear combinations of vectors is equal to s. What’s known about this problem  A variant appeared in a paper of Schmidt in 1907 [T-03]  NP-Hard [N-95]  Applications in signal representation [CW-92,NP-09], amplitude optimization [S-90], function approximation [N-95], and data mining [CRT-06, ZGSD-06, GGIMS-02, GMS-05].  A large number of heuristics have been studied for this problem  Also approximation algorithms under limiting assumptions

Matrix sparsification and the sparse null space problem 13 Another look at min unsatisfy EDR is in fact equivalent to min-unsatisfy ([AK-95] proved one direction) although this seems to have escaped the notice of the sparse approximation theory community. We’ll show how to extend the heuristics and algorithms for EDR (and therefore, min unsatify) for matrix sparsification.

Matrix sparsification and the sparse null space problem 14 Matrix sparsification The following greedy algorithm solves matrix sparsification  We assumes existence of subroutine SIV(A,B), returns the sparsest vector in the span of matrix A that is not in the span of matrix B  Notes: This subroutine can be easily implemented using a heuristic or approximation algorithm for min unsatisfy (see paper) The matrix sparsification algorithm below is a slight simplification (again see paper) Algorithm for matrix sparsification builds matrix B one column at a time  B ← null  For i=n…1 a = SIV(A,B) B ← a [CP-86] proved that the greedy algorithm works for matroids.

Matrix sparsification and the sparse null space problem 15 Algorithms for matrix sparsification We conclude that all algorithms for min unsatisfy (and EDR) apply to matrix sparsification as well.  There exists a randomized Θ(m/log m) approximation algorithm for matrix sparsification.  A large number of heuristics for EDR carry over to matrix sparsification. Practical contribution  The popular l 1 minimization heuristic for EDR carries over to matrix sparsification This heuristic finds a vector v that satisfies Dv=s, while minimizing ||v|| 1 instead of number of non-zeros in v The heuristic is also an approximation algorithm under certain limiting assumptions [F-04]  This heuristic for matrix sparsification has already been implemented since the public posting of our paper!

Matrix sparsification and the sparse null space problem 16 Conclusion Considered the matrix sparsification and sparse null space problems. Showed that they are very hard to approximate. Showed how to extend a large number of studied heuristics and algorithms to these problems