1. Alfredo Nava-Tudela John J. Benedetto, advisor
Sparse Solutions of Linear Systems of Equations and Sparse Modeling of Signals and Images
Alfredo Nava-Tudela
John J. Benedetto, advisor
AMSC 663

2. Outline Problem: Find “sparse solutions” of Ax = b Why do we care?
An application in signal processing: compression
Working definition of “sparse solution”
Why things eventually work, for some cases
How do we find sparse solutions? OMP algorithm
Case study: What happens when we combine the DCT and DWT?
Conclusions/Recap
Project timeline
AMSC 663

3. Problem Let A be an n by m matrix with n < m, and rank(A)=n.
We want to solve
Ax = b, where b is a data or signal vector,
and x is the solution with the fewest number of non-zero entries possible, that is, the “sparsest” one.
Observations:
- A is underdetermined and, since rank(A)=n,
there is an infinite number of solutions. Good!
- How do we find the “sparsest” solution? What does this mean in practice? Is there a unique sparsest solution?
AMSC 663

4. 231 kb, uncompressed, 320x240x3x8 bit
But, why do we care?
We live in a digital media era.
Bandwidths are different and matter: cellphone vs DSL vs Gigabit, etc.
231 kb, uncompressed, 320x240x3x8 bit
74 kb, compressed 3.24:1
JPEG
AMSC 663

5. But, why do we care? 512 x 512 Pixels, 75:1, 10.6 Kbyte 24-Bit RGB,
Ever increasing size of files to transmit demand better compression algorithms.
At the core of the JPEG and JPEG2000 standards, the DCT and the DWT.
Transform encoding
512 x 512 Pixels,
24-Bit RGB,
Size 786 Kbyte
75:1, 10.6 Kbyte
JPEG2000
AMSC 663

6. “Sparsity” equals compression
Both JPEG and JPEG2000 achieve their compression
mainly because at their core one finds a linear transform
(DCT and DWT, respectively) that reduces the number
of non-zero entries required to represent the data, within
an acceptable error.
We can then think of signal compression in terms of our
problem
Ax = b, x is sparse, b is dense, store x!
AMSC 663

7. Working definitions of “sparse”
Convenient to introduce the l0 “norm”:
||x||0 = # {k : xk ≠ 0}
(P0): minx ||x||0 subject to ||Ax - b||2 = 0
(P0e): minx ||x||0 subject to ||Ax - b||2 < e
Observation: Solving (P0) is NP-hard, bummer.
We can cast our problem as a minimization problem
AMSC 663

8. Some theoretical results
Definition: The spark of a matrix A is the minimum
number of linearly dependent columns of A. We write
spark(A) to represent this number.
Theorem: If there is a solution x to Ax = b, and
||x||0 < spark(A) / 2, then x is the sparsest solution.
That is, if y ≠ x also solves the equation, then
||x||0 < ||y||0.
Observation: Computing spark(A) is combinatorial,
therefore hard. Alternative?
Address the question of uniqueness
AMSC 663

9. More theoretical results
Definition: The mutual coherence of a matrix A is the
number
Lemma: spark(A) ≥ 1+1/mu(A).
Theorem: If x solves Ax = b, and ||x||0 < (1+mu(A)-1)/2,
then x is the sparsest solution, as before.
Observation: mu(A) is a lot easier and faster to compute,
but 1+1/mu(A) far worse bound than spark(A), in general.
Mutual coherence gives the worse case scenario criterion for uniqueness
Mention that mutual coherence also plays a role in proving convergence of algorithms, but won’t see this here
AMSC 663

10. Finding sparse solutions:OMP
Orthogonal Matching Pursuit algorithm:
OMP is one of a family of algorithms called “greedy algorithms”.
Goal: implement, validate and test. Matlab.
There are implementations available online against which to compare my implementation.
OMP known to converge when ||x||0 < (1+1/mu(A))/2.
AMSC 663

11. Revisiting compression
Propose to study the compression properties of the
matrix
A = [DCT,DWT]
and compare it with the compression properties of DCT or DWT alone.
Study the behavior of OMP for this problem.
Many wavelet options available to try, e.g., reversible
5/3 or floating-point 9/7 Daubechies, as in JPEG2000.
Interested in compression vs error graph properties.
Matrix A can typically be the concatenation of two basis
AMSC 663

12. Conclusions/Recapitulation
Finding sparse solutions to the linear system of
equations Ax = b, when A is an n by m full rank matrix and n < m, is of interest to the signal processing community.
There are simple criteria to assert the uniqueness of a given sparse solution.
There are algorithms to find sparse solutions, e.g.,
OMP; and their convergence can be guaranteed when there are “sufficiently sparse” solutions.
- Studies on the performance of OMP missing when A is the concatenation of unitary matrices.
AMSC 663

13. Project timeline Oct. 15, 2010: Complete project proposal.
Oct Nov. 5: Implement OMP.
Nov. 8 - Nov. 26: Validate OMP.
Nov Dec. 3: Write mid-year report.
Dec. 6 - Dec. 10: Prepare mid-year oral presentation.
Some time after that, give mid-year oral presentation.
Jan Feb. 11: Testing of OMP. Reproduce paper results.
Jan Apr. 8: Testing of OMP on A = [DCT,DWT].
Apr Apr. 22: Write final report.
Apr Apr. 29: Prepare final oral presentation.
Some time after that, give final oral presentation.
AMSC 663

14. References
A. M. Bruckstein, D. L. Donoho, and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), pp. 34–81.
S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, 1998.
B. K. Natarajan, Sparse approximate solutions to linear systems, SIAM Journal on Computing, 24 (1995), pp
G. W. Stewart, Introduction to Matrix Computations, Academic Press, 1973.
D. S. Taubman and M. W. Mercellin, JPEG 2000: Image Compression Funda- mentals, Standards and Practice, Kluwer Academic Publishers, 2001.
G. K. Wallace, The JPEG still picture compression standard, Communications of the ACM, 34 (1991), pp
AMSC 663