1. Image Priors and the Sparse-Land Model

2. Lets Start with a Virtual Experiment … What are we Expected to See?
Suppose that we take a VERY LARGE set of small images – say that we have accumulated 1e12 patches, each of size 20×20 pixels.
Clearly, every such image is a point in R400. Lets put these point in this 400-dimensional Euclidean space, in the cube [0,1]400.
Now, LETS STEP INTO THIS SPACE and look at the cloud of points we have just generated.
What are we Expected to See?
Deserts! Vast emptiness! Why?
Concentration of points in some regions.
Different densities from one place to another.
Filaments, manifold structure …
In this experiment we have actually created an empirical estimate of the Probability Density Function (PDF)
of … images – Call it P(x)

3. P(x) Answer: EVERYTHING So, Lets Talk about This
We “experimented” with small images, but actually the same phenomena will be found in audio, seismic data, financial data, text-files, … and practically any source of information you are familiar with.
Nevertheless, we will stick to images for the discussion.
Imagine this: a function that can be given an image and return its chances to exist! amazing, No?
Well, what could you do with such a function?
Answer: EVERYTHING

4. Signal/Image Prior P(x)
What is it good for?
Denoising: The measurement is
and we are trying to recover
Region where P(x) is high
Recall that for random noise we have E{(y-x0)Tx0}=0

5. Signal/Image Prior P(x)
What is it good for?
Denoising: The measurement is
and we are trying to recover :
Option 1: MAP
Option 2: MMSE

6. Signal/Image Prior P(x)
What is it good for?
Inverse Problems: The measurement is
and we are trying to recover , as before.
H could be blur, projection, downscaling, subsampling, …

7. Signal/Image Prior P(x)
What is it good for?
Compression: We are given x and a budget of B bits. Our goal is to get the best possible compression (i.e. minimize the error). The approach we take is to divide the whole domain into 2B disjoint sets (Voronoi) and minimize the error w.r.t. the representation vectors (VQ):

8. Signal/Image Prior P(x)
What is it good for?
Sampling: Our goal is to propose sampling and reconstruction strategies, each (or just the first) is parameterized, and optimize the parameters for the smallest possible error:

9. Signal/Image Prior P(x)
What is it good for?
Separation: We are given
Where and are two different signals from two different distributions, and our goal is to separate the signal into its ingredients:

10. Signal/Image Prior P(x)
What is it good for?
Anomaly Detection: We are given x and we are supposed to say if it is an anomaly. This is done by testing

11. Signal/Image Prior P(x)
Question: What is it good for?
Answer: Many great things.
P(x)=?

12. The Evolution of Priors for Images
PDE
smoothness
WLS
Robust stat.
for images
FoE
Transform
GMM,
Co-Sparse Analysis, Low-Rank, …
wavelet
Learn
sparse
Major themes:
L2 → L1 - Linear vs. Non-Linear Approx.
Training on examples
Random generator for x
70’s ’s ’s ’s

13. Signal/Image Prior P(x)
Here is an untold secret:
The vast literature in image processing over the past 4-5 decades is
NOTHING BUT
an evolution of ideas on the identity of P(x), and ways to use it in actual tasks
By the way, the same is true for many other data sources and signals …

14. Linear Versus Non-Linear Approximation
Suppose that our prior is the following (T is unitary):
The matrix  weights the transform elements:
Our goal: Denoising a signal with this prior by solving

15. Linear Versus Non-Linear Approximation
The solution is given by
Implication: We leave the transform coefficients with the small weights and remove the ones with the high weight. The decision who survives the process is fixed by  - This is Linear Approximation

16. Linear Versus Non-Linear Approximation
Suppose now that our prior is the following (T is unitary):
Our goal: Denoising a signal with this prior by solving
We have seen that the solution for this problem is given by soft shrinkage
Implications: Just like before, we filter the signal in the transform domain. This time we leave the dominant coefficients and discard of the small ones. This is known as Nonlinear Approximation.

17. Sparse-Land Signal Generation
Draw k0 – the cardinality of the representation
Draw k0 non-zero values
Draw k0 locations, and generate the representation 
Multiply  by the dictionary D
Add random iid (model) noise e + =
Sparse-Land M
Generator of signals from

18. Sparse-Land vs. Earlier Models
Assume no noise in the model and that D is square and invertible
The Sparse-land model generalizes the previous method by
adopting over-completeness***, and
daring to work with true sparsity and L0
*** What about redundant T? this will be addressed later!

19. This is universally true for signals we operate on
Geometrical Insight
The effective rank d of E (found by SVD) is expected to be very low: d<<n
This is universally true for signals we operate on
The orientation and dimension of this subspace changes form one point to another (smoothly?)

20. Geometrical Insight – Implications
Given a noisy version of x0
How shall we denoise it?
By projecting to the subspace around x0 (chicken and egg)
How come y is not on the subspace itself?
The relative volume of the subspace is negligible
Recall that E{(y-x0)Tx0}=0

21. Geometrical Insight – Denoising in Practice
Given a noisy version of x0
How shall we denoise it?
Non-parametric: Nearest Neighbor (NN), or K-NN
Local-Parametric: Group neighbors, estimate the subspace and project
Parametric: Cluster the DB into K subgroups, and estimate a subspace per each. When a signal is to be denoised, assign it to the closest subgroup, and the project on the corresponding subspace (K=1: PCA)
Sparse-Land: one dictionary encapsulates many such clusters

22. Union of Subspaces (UoS)
We said that with Sparse-Land, one dictionary encapsulates many such clusters
Consider all the signals x that emerge from the same k atoms in D – all of them reside in the same subspace, spanned by these columns.
Thus, every possible support (and there are m-choose-k of them) represent one such subspace which the signal could belong to.
The pursuit task: Given a noisy signal we are searching the “closest subspace” and projecting onto it. It is so hard because of the number of the subspaces involved in this union. =
Sparse-Land

23. Processing Sparse-Land Signals
Objective
Given Data
Goal
Most effective transform - getting the sparsest possible set of iid coefficients
Signal Transform
Cleanest possible signal
Signal Denoising
We have a budget of B bits and we want to best represent the signal
Compress
Treat blur, subsampling, missing values, projection, compressed-sensing
Inverse Problem
The two signals are form different sources and thus have different models
Separate

24. Processing Sparse-Land Signals
Objective
Given Data
Goal
Most effective transform - getting the sparsest possible set of iid coefficients
Signal Transform
Cleanest possible signal
Signal Denoising
We have a budget of B bits and we want to best represent the signal
Compress
Treat blur, subsampling, missing values, projection, compressed-sensing
Inverse Problem
The two signals are form different sources and thus have different models
Separate
All these (and other) processing methods boil down to the solution of
For which we now know that
It is theoretically sensible, and
There are numerical ways to handle it

25. To Summarize
The Sparse-Land forms a general Union of Subspaces, all encapsulated by the concise matrix D.
This follows many earlier work that aims to model signals using a union of subspaces (or mixture of Gaussians – think about it – it is the same).
Sparse-Land is Rooted on solid modeling ideas , while improving on them due to its generality and it solid mathematical foundations