1. Analysis of the Basis Pustuit Algorithm and Applications*
Michael Elad
The Computer Science Department – (SCCM) program
Stanford University
Multiscale Geometric Analysis Meeting
IPAM - UCLA
January 2003
* Joint work with: Alfred M. Bruckstein – CS, Technion
David L. Donoho – Statistics, Stanford

2. Collaborators Freddy Bruckstein Dave Donoho CS Department Technion
Statistics Department Stanford
M.Elad and A.M. Bruckstein, “A Generalized Uncertainty Principle and Sparse Representation in Pairs of Bases", IEEE Trans. On Information Theory, Vol. 48, pp , September 2002.
D. L. Donoho and M. Elad, “Maximal Sparsity Representation via l1 Minimization”, to appear in Proceedings of the Naional Academy of Science. 
Sparse representation and the Basis Pursuit Algorithm

3. General Basis Pursuit algorithm [Chen, Donoho and Saunders, 1995]:
Effective for finding sparse over-complete representations,
Effective for non-linear filtering of signals.
Our work (in progress) – better understanding BP and deploying it in signal/image processing and computer vision applications.
We believe that over-completeness has an important role!
Today we discuss the analysis of the Basis Pursuit algorithm, giving conditions for its success. We then show some stylized applications exploiting this analysis.
Sparse representation and the Basis Pursuit Algorithm

4. Agenda Introduction 2. Two Ortho-Bases 3. Arbitrary dictionary
Previous and current work
2. Two Ortho-Bases
Uncertainty  Uniqueness  Equivalence
3. Arbitrary dictionary
Uniqueness  Equivalence
4. Stylized Applications
Separation of point, line, and plane clusters
5. Discussion
Understanding the BP
Using the BP - Application
Sparse representation and the Basis Pursuit Algorithm

5. Transforms
Define the forward and backward transforms by (assume one-to-one mapping)
s – Signal (in the signal space CN)
 – Representation (in the transform domain CL, LN)
Transforms T in signal and image processing used for coding, analysis, speed-up processing, feature extraction, filtering, …
Sparse representation and the Basis Pursuit Algorithm

6. Atoms from a Dictionary
The Linear Transforms
Special interest - linear transforms (inverse)
Linear
General transforms = L N  s 
Square
In square linear transforms,  is an N-by-N & non-singular.
Atoms from a Dictionary
Unitary
Sparse representation and the Basis Pursuit Algorithm

7. Lack Of Universality
Many available square linear transforms – sinusoids, wavelets, packets, …
Successful transform – one which leads to sparse representations.
Observation: Lack of universality - Different bases good for different purposes.
Sound = harmonic music (Fourier) + click noise (Wavelet),
Image = lines (Ridgelets) + points (Wavelets).
Proposed solution: Over-complete dictionaries, and possibly combination of bases.
Sparse representation and the Basis Pursuit Algorithm

8. + Example – Composed Signal 1 2 3 4 1.0 0.3 0.5
-0.1
-0.05
0.05
0.1 1 2 +
1.0
0.3
0.5
0.05 20 40 60 80
100
120 10
-10 -8 -6 -4 -2
|T{1+0.32}|
DCT Coefficients
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0.1
|T{1+0.32+0.53+0.054}|
0.5 1 3 4 64
128
Sparse representation and the Basis Pursuit Algorithm

9. Spike (Identity) Coefficients
Example – Desired Decomposition 40 80
120
160
200
240 10
-10 -8 -6 -4 -2
DCT Coefficients
Spike (Identity) Coefficients
Sparse representation and the Basis Pursuit Algorithm

10. Matching Pursuit
Given d unitary matrices {k, 1kd}, define a dictionary  = [1, 2 , … d] [Mallat & Zhang (1993)].
Combined representation per a signal s by
Non-unique solution  - Solve for maximal sparsity
Hard to solve – a sub-optimal greedy sequential solver: “Matching Pursuit algorithm” .
Sparse representation and the Basis Pursuit Algorithm

11. Dictionary Coefficients
Example – Matching Pursuit 50
100
150
200
250 10
-10 -8 -6 -4 -2
Dictionary Coefficients
Sparse representation and the Basis Pursuit Algorithm

12. Basis Pursuit (BP)
Facing the same problem, and the same optimization task [Chen, Donoho, Saunders (1995)]
Hard to solve – replace the l0 norm by an l1 : “Basis Pursuit algorithm”
Interesting observation: In many cases it successfully finds the sparsest representation.
Sparse representation and the Basis Pursuit Algorithm

13. Dictionary Coefficients
Example – Basis Pursuit
Dictionary Coefficients
Sparse representation and the Basis Pursuit Algorithm

14. Why ? 2D-Example 0P<1 P=1 P>1
Sparse representation and the Basis Pursuit Algorithm

15. Example – Lines and Points*
Original image
Ridgelets part of the image
Wavelet part of the noisy image
* Experiments from Starck, Donoho, and Candes - Astronomy & Astrophysics 2002.
Sparse representation and the Basis Pursuit Algorithm

16. = + + Example – Galaxy SBS 0335-052* Original Residual Wavelet
Ridgelets
Curvelets = +
* Experiments from Starck, Donoho, and Candes - Astronomy & Astrophysics 2002.
Sparse representation and the Basis Pursuit Algorithm

17. Non-Linear Filtering via BP
Through the previous example – Basis Pursuit can be used for non-linear filtering.
From Transforming to Filtering
What is the relation to alternative non-linear filtering methods, such as PDE based methods (TV, anisotropic diffusion …), Wavelet denoising?
What is the role of over-completeness in inverse problems?
Sparse representation and the Basis Pursuit Algorithm

18. Proving tightness of E-B bounds [Feuer & Nemirovski]
(Our) Recent Work
Proving tightness of E-B bounds [Feuer & Nemirovski]
Improving previous results – tightening the bounds [Elad and Bruckstein]
Proven equivalence between P0 and P1 under some conditions on the sparsity of the representation, and for dictionaries built of two ortho-bases [Donoho and Huo]
Relaxing the notion of sparsity from l0 to lp norm
[Donoho and Elad]
time
1998
1999
2000
2001
2002
Generalized all previous results to any dictionary and Applications [Donoho and Elad]
Sparse representation and the Basis Pursuit Algorithm

19. Before we dive …
Given a dictionary  and a signal s, we want to find the sparse “atom decomposition” of the signal.
Our goal is the solution of
Basis Pursuit alternative is to solve instead
Our focus for now: Why should this work?
Sparse representation and the Basis Pursuit Algorithm

20. Agenda 1. Introduction 2. Two Ortho-Bases 3. Arbitrary dictionary
Previous and current work
2. Two Ortho-Bases
Uncertainty  Uniqueness  Equivalence
3. Arbitrary dictionary
Uniqueness  Equivalence
4. Stylized Applications
Separation of point, line, and plane clusters
5. Discussion N
Sparse representation and the Basis Pursuit Algorithm

21. Our Objective Our Objective is
Given a signal s, and its two representations using  and , what is the lower bound on the sparsity of both?
Our Objective is
We will show that such rule immediately leads to a practical result regarding the solution of the P0 problem.
Sparse representation and the Basis Pursuit Algorithm

22. Mutual Incoherence M – mutual incoherence between  and .
M plays an important role in the desired uncertainty rule.
Properties
Generally,
For Fourier+Trivial (identity) matrices
For random pairs of ortho-matrices
Sparse representation and the Basis Pursuit Algorithm

23. * Uncertainty Rule Theorem 1 Examples: =: M=1, leading to .
* Donoho & Huo obtained a weaker bound
Theorem 1
Examples:
=: M=1, leading to
=I, =FN (DFT): , leading to
Sparse representation and the Basis Pursuit Algorithm

24. Example =I, =FN (DFT) For N=1024, .
The signal satisfying this bound: Picket-fence
200
400
600
800
1000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
Sparse representation and the Basis Pursuit Algorithm

25. Towards Uniqueness
Given a unit norm signal s, assume we hold two different representations for it using 
Thus
Based on the uncertainty theorem we just got:
Sparse representation and the Basis Pursuit Algorithm

26. Uniqueness Rule
In words: Any two different representations of the same
signal CANNOT BE JOINTLY TOO SPARSE. *
* Donoho & Huo obtained a weaker bound
If we found a representation that satisfy
Then necessarily it is unique (the sparsest).
Theorem 2
Sparse representation and the Basis Pursuit Algorithm

27. Uniqueness Implication
We are interested in solving
Somehow we obtain a candidate solution .
The uniqueness theorem tells us that a simple test on
( ) could tell us if it is the solution of P0.
However:
If the test is negative, it says nothing.
This does not help in solving P0.
This does not explain why P1 may be a good replacement.
Sparse representation and the Basis Pursuit Algorithm

28. Equivalence - Goal We are going to solve the following problem
The questions we ask are:
Will the P1 solution coincide with the P0 one?
What are the conditions for such success?
We show that if indeed the P0 solution is sparse enough, then P1 solver finds it exactly.
Sparse representation and the Basis Pursuit Algorithm

29. * Equivalence - Result Given a signal s with a representation ,
Assuming a sparsity on  such that (assume k1<k2)
k1 non-zeros
k2 non-zeros … …
Theorem 3
If k1 and k2 satisfy
then P1 will find the correct solution. *
* Donoho & Huo obtained a weaker bound
A weaker requirement is given by
Sparse representation and the Basis Pursuit Algorithm

30. The Various Bounds Signal dimension: N=1024, Dictionary: =I, =FN ,
Mutual incoherence M=1/32. 4 8 12 16 20 24 28 32 2M 2 K 1
+MK
-1=0 +K
=1/M K 1 2 K 1 +K 2
= /M
Results
Uniqueness: 31 entries and below,
Equivalence:
16 entries and below (D-H),
29 entries and below (E-B). K 1
>K 2 2 K 1
+K = 2
= 0.5(1+1/M) K 4 12 16 20 24 32 8 28 K 1
Sparse representation and the Basis Pursuit Algorithm

31. Equivalence – Uniqueness Gap
For uniqueness we got the requirement
For equivalence we got the requirement
Is this gap due to careless bounding?
Answer [by Feuer and Nemirovski, to appear in IEEE Transactions On Information Theory]: No, both bounds are indeed tight.
Sparse representation and the Basis Pursuit Algorithm

32. Agenda 1. Introduction 2. Two Ortho-Bases 3. Arbitrary dictionary
Previous and current work
2. Two Ortho-Bases
Uncertainty  Uniqueness  Equivalence
3. Arbitrary dictionary
Uniqueness  Equivalence
4. Stylized Applications
Separation of point, line, and plane clusters
5. Discussion L N
Every column is normalized to have an l2 unit norm
Sparse representation and the Basis Pursuit Algorithm

33. Why General Dictionaries?
In many situations
We would like to use more than just two ortho-bases (e.g. Wavelet, Fourier, and ridgelets);
We would like to use non-ortho bases (pseudo-polar FFT, Gabor transform, … ),
In many situations we would like to use non-square transforms as our building blocks (Laplacian pyramid, shift-invariant Wavelet, …).
In the following analysis we assume ARBITRARY DICTIONARY (frame). We show that BP is successful over such dictionaries as well.
Sparse representation and the Basis Pursuit Algorithm

34.  = Uniqueness - Basics v
Given a unit norm signal s, assume we hold two different representations for it using 
In the two-ortho case - simple splitting and use of the uncertainty rule – here there is no such splitting !! = v 
The equation implies a linear combination of columns from  that are linearly dependent. What is the smallest such group?
Sparse representation and the Basis Pursuit Algorithm

35. Uniqueness – Matrix “Spark”
Definition: Given a matrix , define =Spark{} as the smallest integer such that there exists at least one group of  columns from  that is linearly dependent.
Spark =N+1;
Examples:
Spark =2
Sparse representation and the Basis Pursuit Algorithm

36. Generally: 2  =Spark{}  Rank{}+1.
“Spark” versus “Rank”
The notion of spark is confusing – here is an attempt to compare it to the notion of rank
Rank
Definition: Maximal # of columns that are linearly independent
Computation: Sequential - Take the first column, and add one column at a time, performing Gram-Schmidt orthogonalization. After L steps, count the number of non-zero vectors – This is the rank.
Spark
Definition: Minimal # of columns that are linearly dependent
Computation: Combinatorial - sweep through 2L combinations of columns to check linear dependence - the smallest group of linearly dependent vectors is the Spark.
Generally: 2  =Spark{}  Rank{}+1.
Sparse representation and the Basis Pursuit Algorithm

37. Uniqueness – Using the “Spark”
Assume that we know the spark of , denoted by .
For any pair of representations of s we have
By the definition of the spark we know that if v=0 then Thus
From here we obtain the relationship
Sparse representation and the Basis Pursuit Algorithm

38. Uniqueness Rule – 1
Any two different representations of the same signal using an arbitrary dictionary cannot be jointly sparse.
If we found a representation that satisfy
Then necessarily it is unique (the sparsest).
Theorem 4
Sparse representation and the Basis Pursuit Algorithm

39. Lower bound on the “Spark”
Define
(notice the resemblance to the previous definition of M).
We can show (based on Gerśgorin disks theorem) that a lower-bound on the spark is obtained by
Since the Gerśgorin theorem is un-tight, this lower bound on the Spark is too pessimistic.
Sparse representation and the Basis Pursuit Algorithm

40. Uniqueness Rule – 2
Any two different representations of the same signal using an arbitrary dictionary cannot be jointly sparse. *
* This is the same as Donoho and Huo’s bound! Have we lost tightness?
If we found a representation that satisfy
Then necessarily it is unique (the sparsest).
Theorem 5
Sparse representation and the Basis Pursuit Algorithm

41. “Spark” Upper bound The Spark can be found by solving
Use Basis Pursuit
Clearly
Thus
Sparse representation and the Basis Pursuit Algorithm

42. * Equivalence – The Result
Following the same path as shown before for the equivalence theorem in the two-ortho case, and adopting the new definition of M we obtain the following result: *
* This is the same as Donoho and Huo’s bound! Is it non-tight?
Theorem 6
Given a signal s with a representation ,
Assuming that , P1 (BP) is
Guaranteed to find the sparsest solution.
Sparse representation and the Basis Pursuit Algorithm

43. To Summarize so far …
Over-complete linear transforms – great for sparse representations
forward transform?
Basis Pursuit Algorithm
Why works so well?
Practical Implications?
We give explanations (uniqueness and equivalence) true for any dictionary
(a) Design of dictionaries, (b) Test of for optimality, (c) Applications - scrambling, signal separation, inverse problems, …
Sparse representation and the Basis Pursuit Algorithm

44. Agenda 1. Introduction 2. Two Ortho-Bases 3. Arbitrary dictionary
Previous and current work
2. Two Ortho-Bases
Uncertainty  Uniqueness  Equivalence
3. Arbitrary dictionary
Uniqueness  Equivalence
4. Stylized Applications
Separation of point, line, and plane clusters
5. Discussion
Sparse representation and the Basis Pursuit Algorithm

45. Problem Definition Assume a 3D volume of size Voxels.
S contains digital points, lines, and planes, defined algebraically ( ):
Point – one Voxel
Line – p Voxels (defined by a starting Voxel on the cube’s face, and a slope – wrap around is permitted),
Plane – p2 Voxels (defined by a height Voxel in the cube, and 2 slope parameters – wrap around is permitted).
Task: Decompose S into the point, line, and plane atoms – sparse decomposition is desired.
Sparse representation and the Basis Pursuit Algorithm

46. Properties To Use Digital lines and planes are constructed such that:
Any two lines are disjoint, or intersect in a single Voxel.
Any two planes are disjoint, or intersect in a line of p Voxels.
Any digital line intersect a with a digital plane not at all, in a single Voxel, or along the p Voxels being the line itself.
Sparse representation and the Basis Pursuit Algorithm

47. Our Dictionary
We construct a dictionary to contain all occurrences of points, lines, and planes (O(p4) columns).
Every atom in our set is reorganized lexicographically as a column vector of size p3-by-1.
The spark of this dictionary is =p (p points and one line, or p lines and one plane).
The mutual incoherence is given by M=p (normalized inner product of a plane with a line, or line with a point).
Sparse representation and the Basis Pursuit Algorithm

48. Thus we can say that … If we found a representation that satisfies
Due to Theorem 4
If we found a representation that satisfies
then necessarily it is sparsest (P0 solution).
Due to Theorem 6
Given a volume s with a representation
such that , (BP)
is guaranteed to find it.
Sparse representation and the Basis Pursuit Algorithm

49. Implications Assume p=1024:
A representation with fewer than 513 atoms IS KNOWN TO BE THE SPARSEST.
A representation with fewer than 17 atoms WILL BE FOUND BY THE BP.
What happens in between? We know that the second bound is very loose! BP will succeed far above 17 atoms. Further work is needed to establish this fact.
What happens above 513 atoms? No theoretical answer yet!
Sparse representation and the Basis Pursuit Algorithm

50. Towards Practical Application
The algebraic definition of the points, lines, and planes atoms was chosen to enable the computation of  and M.
Although wrap around makes this definition non-intuitive, these results are suggestive of what may be the case for geometrically intuitive definitions of points, lines, and planes.
Further work is required to establish this belief.
Sparse representation and the Basis Pursuit Algorithm

51. Agenda 1. Introduction 2. Two Ortho-Bases 3. Arbitrary dictionary
Previous and current work
2. Two Ortho-Bases
Uncertainty  Uniqueness  Equivalence
3. Arbitrary dictionary
Uniqueness  Equivalence
4. Stylized Applications
Separation of point, line, and plane clusters
5. Discussion
Sparse representation and the Basis Pursuit Algorithm 51

52. Summary The Basis Pursuit can be used as a
Forward transform, leading to sparse representation.
Way to achieve non-linear filtering.
The dream: the over-completeness idea is highly effective, and should be used in modern methods in representation and inverse-problems.
We would like to contribute to this development by
Supplying clear(er) explanations about the BP behavior,
Improve the involved numerical tools, and then
Deploy it to applications.
Sparse representation and the Basis Pursuit Algorithm 53

53. Future Work What dictionary to use? Relation to learning?
BP beyond the bounds – Can we say more?
Relaxed notion of sparsity? When zero is really zero?
How to speed-up BP solver (both accurate and approximate)?
Theory behind approximate BP?
Applications – Demonstrating the concept for practical problems, such as denoising, coding, restoration, signal separation …
Sparse representation and the Basis Pursuit Algorithm 54