1. Compressed Sensing & Sparse Signal Recovery
Trần Duy Trác
ECE Department
The Johns Hopkins University
Baltimore, MD 21218

2. Outline Compressed Sensing: Quick Overview
Motivations. Toy Examples
Incoherent Bases and Restrictive Isometry Property
Decoding Strategy
L0 versus L1 versus L2
Basis Pursuit and Matching Pursuit
Compressed Sensing in Video Processing
2D Separable Measurement Ensemble (SME)
Distributed compressed video sensing (DISCOS)
Layered compressed sensing for robust video transmission

3. Compressed Sensing History
Emmanuel Candès and Terence Tao, ”Decoding by linear programming” IEEE Trans. on Information Theory, 51(12), pp , December 2005
Emmanuel Candès, Justin Romberg, and Terence Tao, ”Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. on Information Theory, 52(2) pp , Feb
David Donoho, ”Compressed sensing,” IEEE Trans. on Information Theory, 52(4), pp , Apr
Emmanuel Candès and Michael Wakin, ”An introduction to compressive sampling,” IEEE Signal Processing Magazine, 25(2), pp , Mar

4. Traditional Data Acquisition: Sampling
Shannon Sampling Theorem
In order for a band-limited signal x(t) to be reconstructed
perfectly, it must be sampled at rate t
x(t) ^ t
x(t)

5. Traditional Compression Paradigm
Receive
Decompress
Sample
Compress
Transmit/Store
MP3, JPEG, JPEG200, MPEG…
samples
largest coefficients
Sample first and then worry about compression later!

6. Sparse Signals Digital signals in practice are often sparse
basis functions
largest coefficients
transform coefficients
Digital signals in practice are often sparse
Audio: MP3, AAC… ~10:1 compression
Images: JPEG, JPEG2000… ~20:1 compression
Video sequences: MPEG2, MPEG4… ~40:1 compression

7. Sparse Signals II basis functions N-pixel image transform coefficients
nonzero entries
: -sparse signal in non-sparse domain
: sparse signal

8. Definition & Notation N = length of signal x
K = the sparsity level of x or x is called K-sparse
M = the number of measurements (samples) taken at the encoder

9. Compressed Sensing Framework
Encoding: obtain M measurements y from linear projection onto an incoherent basis
Sensing matrix =
has only K nonzero entries
Compressed measurements
Decoding: reconstruct x from measurements y via nonlinear optimization with sparsity prior

10. At Encoder: Signal Sensing
y is not sparse, looks iid
Random projection works well!
Sensing & sparsifying matrix must be incoherent
Each measurement y contains a little information of each sample of x

11. At Decoder: Signal Reconstruction
Recover x from the set of measurements y
Without the sparseness assumption, the problem is ill-posed
With sparseness assumption, the L0-norm minimization problem is well-posed but computationally intractable
With sparseness assumption, the L1-norm minimization can be solved via linear programming – Basis Pursuit!

12. L0- and L1-norm Reconstruction
- norm reconstruction : take advantage of sparsity prior
We find the sparsest solution
Problem: Combinatorial searching
Exhaustive computation
- norm reconstruction : Compressed sensing framework
This is a convex optimization problem
Using linear programming to solve
Also can find the sparsest which turns out to be the exact solution
- 12 -

13. L1-Minimization Problem Let Standard LP Many available techniques
Simplex, primal-dual interior-point, log-barrier…

14. CS Reconstruction: Matching Pursuit
Problem
Basis Pursuit
Greedy Pursuit – Iterative Algorithms
At each iteration, try to identify columns of A (atoms) that are associated with non-zero entries of x
significant atoms y A x

15. Matching Pursuit
MP: At each iteration, MP attempts to identify the most significant atom. After K iteration, MP will hopefully identify the signal!
, set residual vector , selected index set
Find index yielding the maximal correlation with the residue
Augment selected index set:
Update the residue:
, and stop when
t = K

16. Orthogonal Matching Pursuit
OMP: guarantees that the residue is orthogonal to all previously chosen atoms  no atom will be selected twice!
, set residual vector , index set
Find index that yields the maximal correlation with residue
Augment
Find new signal estimate by solving
Set the new residual :
, and stop when
t = K

17. Subspace Pursuit
SP: pursues the entire subspace that the signal lives in at every iteration steps and adds a backtracking mechanism!
Initialization
Selected set
Signal estimate
Residue
, go to Step 2; stop when residue energy does not decrease anymore

18. Incoherent Bases: Definition
Suppose signal is sparse in a orthonomal transform domain
Take K measurements from an orthonormal sensing matrix
Definition : Coherence between and T
With ,

19. Incoherent Bases: Properties
Bound of coherence :
When is small, we call 2 bases are incoherent
Intuition: when 2 bases are incoherent, all entries of matrix are spread out each measurement will contains more information of signal we hope to have small
Some pair of incoherent bases :
DFT and identity matrix :
Gaussian (Bernoulli) matrix and any other basis:

20. Universality of Incoherent Bases
Random Gaussian white noise basis is incoherent with any fixed orthonormal basis with high probability
If the signal is sparse in frequency, the sparsifying matrix is
Product of is still Gaussian white noise!

21. Restricted Isometry Property
non-zero entries
Sufficient condition for exact recovery: All sub-matrices composed of columns are nearly orthogonal

22. L2-norm Reconstruction
- norm reconstruction : classical approach
We find
with smallest energy
Closed-form solution
Unfortunately, this method almost never find the sparsest and correct answer

23. Why Is L1 Better Than L2? The line intersect
Bad point
The line
intersect
circle at a non-sparse point
The line
intersect
diamond at the sparse point
Unique and exact solution
- 23 -

24. How Many Measurements Are Enough?
Theorem (Candes, Romberg and Tao) : Suppose x has support on T , M rows of matrix F is selected uniformly at random from N rows of DFT matrix N x N, then if M obeying :
Minimize L1 will recover x exactly with extremely high probability :
In practice:
is enough to perfectly recover

25. One-Pixel Compressed Sensing Camera
Courtesy of Richard Baraniuk & Kevin Rice

26. CS Analog-to-Digital Converter

27. Modulated Wideband Converter

28. Imaging from a Random Lens
Not only possible but also yields better quality when there are a large percentage of missing samples
Can under-sample when the scene is incoherent

29. Compressed Sensing in Medical Imaging
Goal so far:
Achieve faster MR imaging while maintaining reconstruction quality
Methods:
under sample discrete Fourier space using some pseudo-random patterns
then reconstruct using L1-minimization (Lustig) or homotopic L0-minimization (Trzasko)

30. Sparsity of MR Images Brain MR images, cardiology dynamic MR images
Sparse in Wavelet, DCT domains
Not sparse in spatial domain, or finite-difference domain
Can be reconstructed with good quality using 5-10% of coefficients
Angiogram images
Space in finite-difference domain and spatial domain
(edges of blood vessels occupy only 5% in space)
allow very good Compressed Sensing performance

31. Sampling Methods Using smooth k-space trajectories
Cartesian scanning
Radial scanning
Spiral scanning
Fully sampling in each read-out
Under sampling by:
Cartesian grid: under sampling in phase encoding (uniform, non-uniform)
Radial: angular under-sampling (using less angles)
Spiral: using less spirals and randomly perturb spiral trajectories

32. Sampling Patterns: Spiral & Radial
Spiral scanning: uniform density, varying density, and perturbed spiral trajectories
New algorithm (FOCUSS) allows reconstruction without angular aliasing artifacts 32

33. Reconstruction Methods
Lustig’s : L1-minimization with non-linear Conjugate Gradient method
Trzasko’s: homotopic L0-minimization
MIP: maximum intensity projection 33

34. Reconstruction Results (2DFT)
Multi-slice 2DFT fast spin echo CS at 2.4 acceleration.

35. Results – 3DFT
Contrast-enhanced 3D angiography reconstruction results as a function of acceleration.
Left Column: Acceleration by LR. Note the diffused boundaries with acceleration. Middle Column:
ZF-w/dc reconstruction. Note the increase of apparent noise with acceleration. Right Column: CS
reconstruction with TV penalty from randomly under-sampled k-space

36. Results – Radial scan, FOCUSS reconstruction1 3 2 4 1
Results – Radial scan, FOCUSS reconstruction
Reconstruction results from full-scan with uniform angular sampling between 0◦–360◦.
1st row: Reference reconstruction from 190 views.
2nd row: Reconstruction results from 51 views using LINFBP.
3rd row: Reconstruction results from 51 views using CG-ALONE.
4th row: Reconstruction results from 51 views using PR-FOCUSS 36

37. Results – spiral (a) Sagittal T2-weighted image of the spine,
(b) simulated k-space trajectory (multishot Cartesian spiral, 83% under-sampling),
(c) minimum energy
solution via zero-filling,
(d) reconstruction by L1 minimization,
(e) reconstruction by homotopic L0 minimization using (|∇u|,  ) = |∇u|/ (|∇u|+ ),
(f) line profile across C6, (g-j) enlargements of (a,c-e), respectively.

38. Practical Sensing Matrix Design
Compressed measurement
Sensing matrix = Φ y x 38 38 38

39. Practical Sensing Matrix Design
Sparsifying transform
Sparse transform coefficients  Ψ
Compressed measurement
Sensing matrix = Φ y
* = Argmin ||1 s.t. y = ΦΨ
Sparse Signal Recovery
x* = Ψ* 39 39 39

40. Structurally Random Matrix (SRM)
Random down-sampler
Local randomizer
Fast transform
FFT, WHT, DCT,… D F R
Use a different d_{ii} for the randommizer and the downsampler 40 40 40 40

41. Structurally Random Matrix (SRM)
Random downsampler
Global randomizer
Uniformly random permutation matrix
Fast transform
FFT, WHT, DCT,… D F R 41 41 41 41

42. Performance Analysis Theorem [Do, Lu, Nguyen & Tran]
Assume a signal x is K-sparse,
y=Ax
where A=DFR is a SRM.
If M ~O(KlogN), with high prob., x can be perfectly recovered from y via a l1 minimization reconstruction
Proof Techniques
Central Limit Theorem (CLT)
Combinatorial CLT
Concentration Inequalities
Exact Recovery Principle[E. Candes]
Do, Lu, Nguyen & Tran, Fast Compressive Sensing using Structurally Random Matrices, (Proceeding of ICASSP, pp , April, 2008)
Emmanuel Candès and Justin Romberg, Sparsity and incoherence in compressive sampling. (Inverse Problems, 23(3) pp , 2007) 42 42 42

43. Structurally random matrices Completely random matrices (Gaussian)
Feature Comparison
Features
Structurally random matrices
Completely random matrices (Gaussian)
No. of measurements for exact recovery
O(KlogN)
Sensing complexity
NlogN
O(KNlogN)
Implementation in hardware & optics
Very easy
Difficult
Fast computability
Yes No
Block processing 1 -1 43 43

44. Linear Regression noise or innovation new observation
linear combination of past collected data

45. Sparse Model for Concealment
• • •
Dij
ij
+ eij
noise
yij
missing portion
reference frame
corrupted frame

46. Video De-noising
noise
Clean key frame xB
• • • DB B
+ eB 46 46 46

47. Video Denoising: Sparse Model
xB = DB B + eB
= [DB| IB] B eB
Noisy Block
Identity matrix
= WB B
*B = Argmin |B|1 s.t. xB = WB B
x*B = DB*B
Denoised Block
Sparsity constrained block prediction 47 47 47

48. Video Super-Resolution
High resolution key frame
Noisy low resolution non-key frames
Unobservable high resolution non-key frame SB HB =
+ eB
subsampling
bluring yB xB
noise
Typical relationship between LR and HR patches 48 48 48

49. Video Super-Resolution: Sparse Model
Dictionary of neighboring blocks
xB= SBHByB + eB = SBHBDB B + eB
= [SBHBDB| IB] B eB
= WB B
*B = Argmin |B|1 s.t. xB = WB B
y*B = DB*B
High resolution block approximation
Sparsity constrained block prediction 49 49 49

50. Blocking-Effect Elimination
Averaging from multiple approximations from shifted and overlapping grids 50 50 50

51. Joint Registration & Anomaly Detection
reference image X
test image Y M N …

52. Change Detection for SAR Images
Reference Image Test Image L1-norm Map Change Detection

53. Conclusion Compressed sensing Sparse recovery
A different paradigm for data acquisition
Low sampling resolution in incoherent domain
Sample less and compute more
Simple encoding; most computation at decoder
Exploit a priori signal sparsity at decoder
Universality
Robustness: CS measurements are democratic!
Sparse recovery
Still spot the common trend given incomplete & inaccurate data samples
Detection, classification, recognition
Denoising, concealment, enhancement

54. References http://www.dsp.ece.rice.edu/cs/