1. Mathematics and Image Analysis, MIA'06
Sparse & Redundant Signal Representation and its Role in Image Processing
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Mathematics and Image Analysis, MIA'06
Paris, September , 2006

2. Sparsity and Redundancy
Today’s Talk is About
Sparsity and Redundancy
We will try to show today that
Sparsity & Redundancy can be used to design new/renewed & powerful signal/image processing tools.
We will present both the theory behind sparsity and redundancy, and how those are deployed to image processing applications.
This is an overview lecture, describing the recent activity in this field.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

3. Agenda Welcome to Sparseland A Visit to Sparseland
Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations
How & why should this work?
3. Problem 2: What About D?
The quest for the origin of signals
4. Problem 3: Applications
Image filling in, denoising, separation, compression, …
Welcome to Sparseland
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

4. M Generating Signals in SparselandK N
A fixed Dictionary
Every column in D (dictionary) is a prototype signal (Atom). M
The vector  is generated randomly with few non-zeros in random locations and random values.
A sparse & random vector N
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

5. M Sparseland Signals Are Special
Simple: Every generated signal is built as a linear combination of few atoms from our dictionary D
Rich: A general model: the obtained signals are a special type mixture-of-Gaussians (or Laplacians).
Multiply by D M
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

6. M T Transforms in Sparseland ? M
Assume that x is known to emerge from
We desire simplicity, independence, and expressiveness. M
How about “Given x, find the α that generated it in ” ? T
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

7. So, In Order to Transform …
We need to solve an under-determined linear system of equations:
Known
Among all (infinitely many) possible solutions we want the sparsest !!
We will measure sparsity using the L0 norm:
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

8. Measure of Sparsity? -1 +1 1
As p  0 we get a count of the non-zeros in the vector
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

9. M Signal’s Transform in Sparseland Multiply by D
A sparse & random vector M
Are there practical ways to get ?
4 Major Questions
Is ? Under which conditions?
How effective are those ways?
How would we get D?
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

10. M Q Inverse Problems in Sparseland ? M
Assume that x is known to emerge from . M M
Suppose we observe , a “blurred” and noisy version of x with How will we recover x?
Noise
How about “find the α that generated the x …” again? Q
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

11. 4 Major Questions (again!)
Inverse Problems in Sparseland ? M
4 Major Questions (again!)
How would we get D?
Are there practical ways to get ?
How effective are those ways?
Is ? How far can it go?
Multiply by D
A sparse & random vector
“blur” by H
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

12. Sparseland is HERE Back Home … Any Lessons?
Several recent trends worth looking at:
JPEG to JPEG From (L2-norm) KLT to wavelet and non-linear approximation
Sparseland is HERE
Sparsity.
From Wiener to robust restoration – From L2-norm (Fourier) to L1. (e.g., TV, Beltrami, wavelet shrinkage …)
Sparsity.
From unitary to richer representations – Frames, shift-invariance, steerable wavelet, contourlet, curvelet
Overcompleteness.
Approximation theory – Non-linear approximation
Sparsity & Overcompleteness.
ICA and related models
Independence and Sparsity.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

13. To Summarize so far …
The Sparseland model for signals is very interesting
Who cares?
We do! it is relevant to us
So, what are the implications?
We need to answer the 4 questions posed, and then show that all this works well in practice
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

14. T Q Agenda 1. A Visit to Sparseland
Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations
How & why should this work?
3. Problem 2: What About D?
The quest for the origin of signals
4. Problem 3: Applications
Image filling in, denoising, separation, compression, … T Q
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

15. Lets Start with the Transform …
Our dream for Now: Find the sparsest solution of
Known
known
Put formally,
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

16. Suppose we can solve this exactly
Question 1 – Uniqueness?
Multiply by D M
Suppose we can solve this exactly
Why should we necessarily get ?
It might happen that eventually
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

17. Matrix “Spark”
* In tensor decomposition, Kruskal defined something similar already in 1989. *
Donoho & E. (‘02)
Definition: Given a matrix D, =Spark{D} is the smallest and and number of columns that are linearly dependent.
Example:
Spark = 3
Rank = 4
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

18. Uniqueness Rule M Suppose this problem has been solved somehow
If we found a representation that satisfy
Then necessarily it is unique (the sparsest).
Uniqueness
Donoho & E. (‘02)
This result implies that if generates signals using “sparse enough” , the solution of the above will find it exactly. M
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

19. M Question 2 – Practical P0 Solver?
Are there reasonable ways to find ?
Multiply by D
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

20. Matching Pursuit (MP)
Mallat & Zhang (1993)
The MP is a greedy algorithm that finds one atom at a time.
Step 1: find the one atom that best matches the signal.
Next steps: given the previously found atoms, find the next one to best fit …
The Orthogonal MP (OMP) is an improved version that re-evaluates the coefficients after each round.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

21. Basis Pursuit (BP) Instead of solving Solve Instead
Chen, Donoho, & Saunders (1995)
Instead of solving
Solve Instead
The newly defined problem is convex (linear programming).
Very efficient solvers can be deployed:
Interior point methods [Chen, Donoho, & Saunders (`95)] ,
Sequential shrinkage for union of ortho-bases [Bruce et.al. (`98)],
Iterated shrinkage [Figuerido & Nowak (`03), Daubechies, Defrise, & Demole (‘04), E. (`05), E., Matalon, & Zibulevsky (`06)].
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

22. M Question 3 – Approx. Quality? How effective are the MP/BP
Multiply by D
How effective are the MP/BP
in finding ?
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

23. Assume normalized columns
Evaluating the “Spark”
Compute DT = D
DTD
Assume normalized columns
The Mutual Coherence M is the largest off-diagonal entry in absolute value.
The Mutual Coherence is a property of the dictionary (just like the “Spark”). In fact, the following relation can be shown:
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

24. BP and MP Equivalence Equivalence
Given a signal x with a representation ,
Assuming that , BP and MP are
Guaranteed to find the sparsest solution.
Donoho & E. (‘02)
Gribonval & Nielsen (‘03)
Tropp (‘03)
Temlyakov (‘03)
MP and BP are different in general (hard to say which is better).
The above result corresponds to the worst-case.
Average performance results are available too, showing much better bounds [Donoho (`04), Candes et.al. (`04), Tanner et.al. (`05), Tropp et.al. (`06), Tropp (’06)].
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

25. What About Inverse Problems?
“blur” by H
Multiply by D
A sparse & random vector
We had similar questions regarding uniqueness, practical solvers, and their efficiency.
It turns out that similar answers are applicable here due to several recent works [Donoho, E. and Temlyakov (`04), Tropp (`04), Fuchs (`04), Gribonval et. al. (`05)].
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

26. To Summarize so far …
The Sparseland model is relevant to us. We can design transforms and priors based on it
Find the sparsest solution?
Use pursuit Algorithms
Why works so well?
What next?
A sequence of works during the past 3-4 years gives theoretic justifications for these tools behavior
How shall we find D?
Will this work for applications? Which?
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

27. Agenda 1. A Visit to Sparseland
Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations
How & why should this work?
3. Problem 2: What About D?
The quest for the origin of signals
4. Problem 3: Applications
Image filling in, denoising, separation, compression, …
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

28. M Problem Setting Multiply by D
Given these P examples and a fixed size [NK] dictionary D, how would we find D?
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

29. Uniqueness? M If is rich enough* and if Uniqueness then D is unique.
Aharon, E., & Bruckstein (`05) M
“Rich Enough”: The signals from could be clustered to groups that share the same support. At least L+1 examples per each are needed.
Comments:
This result is proved constructively, but the number of examples needed to pull this off is huge – we will show a far better method next.
A parallel result that takes into account noise is yet to be constructed.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

30. X A D  Practical Approach – Objective
Each example is a linear combination of atoms from D
Each example has a sparse representation with no more than L atoms
Field & Olshausen (96’)
Engan et. al. (99’)
Lewicki & Sejnowski (00’)
Cotter et. al. (03’)
Gribonval et. al. (04’)
(n,K,L are assumed known, D has norm. columns)
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

31. Column-by-Column by Mean computation over the relevant examples
K–Means For Clustering
Clustering: An extreme sparse coding D
Initialize D
Sparse Coding
Nearest Neighbor XT
Dictionary Update
Column-by-Column by Mean computation over the relevant examples
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

32. Column-by-Column by SVD computation
The K–SVD Algorithm – General D
Initialize D
Sparse Coding
Use MP or BP X T
Dictionary Update
Column-by-Column by SVD computation
Aharon, E., & Bruckstein (`04)
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

33. For the jth example we solve
K–SVD Sparse Coding Stage D
For the jth example we solve X T
Pursuit Problem !!!
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

34. D K–SVD Dictionary Update Stage
Gk : The examples in and that use the column dk. D
The content of dk influences only the examples in Gk.
Let us fix all A apart from the kth column and seek both dk and the kth column to better fit the residual!
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

35. D K–SVD Dictionary Update Stage We should solve: ResidualE
dk is obtained by SVD on the examples’ residual in Gk.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

36. D D K–SVD: A Synthetic Experiment Results
Create A 2030 random dictionary with normalized columns
Generate 2000 signal examples with 3 atoms per each and add noise D
Train a dictionary using the KSVD and MOD and compare
Results
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

37. To Summarize so far …
The Sparseland model for signals is relevant to us. In order to use it effectively we need to know D
How D can be found?
Use the K-SVD algorithm
Will it work well?
What next?
We have shown how to practically train D using the K-SVD
Show that all the above can be deployed to applications
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

38. Agenda 1. A Visit to Sparseland
Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations
How & why should this work?
3. Problem 2: What About D?
The quest for the origin of signals
4. Problem 3: Applications
Image filling in, denoising, separation, compression, …
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

39. = Application 1: Image Inpainting
Assume: the signal x has been created by x=Dα0 with very sparse α0.
Missing values in x imply missing rows in this linear system.
By removing these rows, we get
Now solve
If α0 was sparse enough, it will be the solution of the above problem! Thus, computing Dα0 recovers x perfectly. =
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

40. Application 1: Side Note
Compressed Sensing is leaning on the very same principal, leading to alternative sampling theorems.
Assume: the signal x has been created by x=Dα0 with very sparse α0.
Multiply this set of equations by the matrix Q which reduces the number of rows.
The new, smaller, system of equations is
If α0 was sparse enough, it will be the sparsest solution of the new system, thus, computing Dα0 recovers x perfectly.
Compressed sensing focuses on conditions for this to happen, guaranteeing such recovery.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

41. Application 1: The Practice
Given a noisy image y, we can clean it using the Maximum A-posteriori Probability estimator by [Chen, Donoho, & Saunders (‘95)].
What if some of the pixels in that image are missing (filled with zeros)? Define a mask operator as the diagonal matrix W, and now solve instead
When handling general images, there is a need to concatenate two dictionaries to get an effective treatment of both texture and cartoon contents – This leads to separation [E., Starck, & Donoho (’05)].
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

42. Inpainting Results
Source
Predetermined dictionary: Curvelet (cartoon) + Overlapped DCT (texture)
Outcome
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

43. Inpainting Results
More about this application will be given in Jean-Luc Starck talk that follows.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

44. Application 2: Image Denoising
Given a noisy image y, we have already mentioned the ability to clean it by solving
When using the K-SVD, it cannot train a dictionary for large support images – How do we go from local treatment of patches to a global prior?
The solution: Force shift-invariant sparsity - on each patch of size N-by-N (e.g., N=8) in the image, including overlaps.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

45. Our MAP penalty becomes
Application 2: Image Denoising
Our MAP penalty becomes
Our prior
Extracts a patch in the ij location
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

46. Application 2: Image Denoising
The dictionary (and thus the image prior) is trained on the corrupted itself!
This leads to an elegant fusion of the K-SVD and the denoising tasks.
x=y and D known
x and ij known
D and ij known
Compute ij per patch
using the matching pursuit
K-SVD
Compute D to minimize
using SVD, updating one column at a time
Compute x by
which is a simple averaging of shifted patches
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

47. D Application 2: The Algorithm
Sparse-code every patch of 8-by-8 pixels D
Update the dictionary based on the above codes
The computational cost of this algorithm is mostly due to the OMP steps done on each image patch - O(N2×L×K×Iterations) per pixel.
Compute the output image x
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

48. Denoising Results Source
The obtained dictionary after 10 iterations
Initial dictionary (overcomplete DCT) 64×256
Result dB
The results of this algorithm tested over a corpus of images and with various noise powers compete favorably with the state-of-the-art - the GSM+steerable wavelets denoising algorithm by [Portilla, Strela, Wainwright, & Simoncelli (‘03)], giving ~1dB better results on average.
Noisy image
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

49. Application 3: Compression
The problem: Compressing photo-ID images.
General purpose methods (JPEG, JPEG2000) do not take into account the specific family.
By adapting to the image-content (PCA/K-SVD), better results could be obtained.
For these techniques to operate well, train dictionaries locally (per patch) using a training set of images is required.
In PCA, only the (quantized) coefficients are stored, whereas the K-SVD requires storage of the indices as well.
Geometric alignment of the image is very helpful and should be done.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

50. Application 3: The Algorithm
On the training set
Detect main features and warp the images to a common reference (20 parameters)
Training set (2500 images)
Divide the image into disjoint 15-by-15 patches. For each compute mean and dictionary
Per each patch find the operating parameters (number of atoms L, quantization Q)
Warp, remove the mean from each patch, sparse code using L atoms, apply Q, and dewarp
On the test image
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

51. Compression Results Results for 820 Bytes per each file 11.99 10.83
10.93
10.49
8.92
8.71
8.81
7.89
8.61
5.56
4.82
5.58
Results for 820 Bytes per each file
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

52. Compression Results Results for 550 Bytes per each file 15.81 14.67
15.30
13.89
12.41
12.57
10.66
9.44
10.27
6.60
5.49
6.36
Results for 550 Bytes per each file
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

53. ? Compression Results Results for 400 Bytes per each file 18.62 16.12
16.81
12.30
11.38
12.54
7.61
6.31
7.20
Results for 400 Bytes per each file
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

54. Today We Have Discussed
1. A Visit to Sparseland
Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations
How & why should this work?
3. Problem 2: What About D?
The quest for the origin of signals
4. Problem 3: Applications
Image filling in, denoising, separation, compression, …
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

55. There are difficulties in using them!
Summary
Sparsity and Over- completeness are important ideas. Can be used in designing better tools in signal/image processing
There are difficulties in using them!
This is an on-going work:
Deepening our theoretical
understanding,
Speedup of pursuit methods,
Training the dictionary,
Demonstrating applications, …
The dream?
Future transforms and regularizations should be data-driven, non-linear, overcomplete, and promoting sparsity.
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing

56. More information, including these slides, can be found in my web-page
Thank You for Your Time
More information, including these slides, can be found in my web-page
THE END !!
Sparse and Redundant
Signal Representation,
and Its Role in
Image Processing