1. Compressive Sensing A New Approach to Image Acquisition and Processing
Richard Baraniuk
Mona Sheikh
Rice University

2. The Digital Universe Size: 281 billion gigabytes generated in 2007
digital bits > stars in the universe
growing by a factor of 10 every 5 years
> Avogadro’s number (6.02x1023) in 15 years
Growth fueled by multimedia data
audio, images, video, surveillance cameras, sensor nets, … (2B fotos on Flickr, 4.2M security cameras in UK, …)
In 2007 digital data generated > total storage
by 2011, ½ of digital universe will have no home
[Source: IDC Whitepaper “The Diverse and Exploding Digital Universe” March 2008]

3. Act I. What’s wrong with today’s. multimedia sensor systems
Act I What’s wrong with today’s multimedia sensor systems? why go to all the work to acquire massive amounts of multimedia data only to throw much/most of it away?
Act II One way out: dimensionality reduction (compressive sensing) enables the design of radically new sensors and systems
Act III Compressive sensing in action new cameras, imagers, ADCs, …
Postlude Open-access education

4. Sense by Sampling sample
Example: Large Hadron Collider will produce 10 peta bytes of data per second; need to triage that down to 100/sec “potentially interesting events” in real time

5. Sense by Sampling too sample much data!
Example: Large Hadron Collider will produce 10 peta bytes of data per second; need to triage that down to 100/sec “potentially interesting events” in real time

6. Sense then Compress
sample
compress
JPEG
JPEG2000 …
decompress

7. Sparsity
large wavelet coefficients
(blue = 0)
pixels

8. Sparsity large wavelet coefficients pixels wideband signal samples
(blue = 0)
pixels
wideband signal samples
large Gabor (TF) coefficients
frequency
time

9. What’s Wrong with this Picture?
Why go to all the work to acquire N samples only to discard all but K pieces of data?
sample
compress
decompress

10. What’s Wrong with this Picture?
linear processing
linear signal model
(bandlimited subspace)
nonlinear processing
nonlinear signal model
(union of subspaces)
sample
compress
decompress

11. Compressive Sensing
Directly acquire “compressed” data via dimensionality reduction
Replace samples by more general “measurements”
compressive sensing
recover

12. Compressive Sensing Theory A Geometrical Perspective

13. Sampling Signal is -sparse in basis/dictionary
WLOG assume sparse in space domain
sparse signal
nonzero entries

14. Sampling Signal is -sparse in basis/dictionary Sampling
WLOG assume sparse in space domain
Sampling
sparse signal
measurements
nonzero entries

15. Compressive Sensing
When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction
sparse signal
measurements
nonzero
entries

16. How Can It Work?
Projection not full rank… … and so loses information in general
Ex: Infinitely many ’s map to the same (null space)

17. How Can It Work?
Projection not full rank… … and so loses information in general
But we are only interested in sparse vectors
columns

18. How Can It Work?
Projection not full rank… … and so loses information in general
But we are only interested in sparse vectors
is effectively MxK
columns

19. How Can It Work?
Projection not full rank… … and so loses information in general
But we are only interested in sparse vectors
Design so that each of its MxK submatrices are full rank (ideally close to orthobasis)
Restricted Isometry Property (RIP)
columns

20. Geometrical Interpretation of RIP
An information preserving projection preserves the geometry of the set of sparse signals
sparse signal
nonzero
entries 20

21. Geometrical Interpretation of RIP
An information preserving projection preserves the geometry of the set of sparse signals
Model: union of K-dimensional subspaces aligned w/ coordinate axes (highly nonlinear!)
sparse signal
nonzero
entries 21

22. RIP = Stable Embedding
An information preserving projection preserves the geometry of the set of sparse signals
RIP ensures that
K-dim subspaces 22

23. RIP = Stable Embedding
An information preserving projection preserves the geometry of the set of sparse signals
RIP ensures that 23

24. How Can It Work?
Projection not full rank… … and so loses information in general
Design so that each of its MxK submatrices are full rank (RIP)
Unfortunately, a combinatorial, NP-complete design problem
columns

25. Insight from the 70’s [Kashin, Gluskin]
Draw at random
iid Gaussian
iid Bernoulli …
Then has the RIP with high probability provided
columns

26. Insight from the 70’s [Kashin, Gluskin]
Draw at random
iid Gaussian
iid Bernoulli …
Then has the RIP with high probability provided
columns

27. Randomized Sensing
Measurements = random linear combinations of the entries of
No information loss for sparse vectors whp
sparse signal
measurements
nonzero
entries

28. CS Signal Recovery Goal: Recover signal from measurements
Problem: Random projection not full rank (ill-posed inverse problem)
Solution: Exploit the sparse/compressible geometry of acquired signal

29. CS Signal Recovery Random projection not full rank
Recovery problem: given find
Null space
Search in null space for the “best” according to some criterion
ex: least squares
(N-M)-dim hyperplane at random angle

30. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Closed-form solution:

31. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Closed-form solution:
Wrong answer! 31

32. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Closed-form solution:
Wrong answer! 32

33. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
“find sparsest vector in translated nullspace” 33

34. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Correct!
But NP-Complete alg
“find sparsest vector in translated nullspace” 34

35. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Convexify the optimization
Candes Romberg Tao
Donoho 35

36. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Convexify the optimization
Correct!
Polynomial time alg (linear programming) 36

37. Compressive Sensing
sparse signal
random measurements
nonzero
entries
Signal recovery via optimization [Candes, Romberg, Tao; Donoho] 37

38. Compressive Sensing Signal recovery via iterative greedy algorithm
sparse signal
random measurements
nonzero
entries
Signal recovery via iterative greedy algorithm
(orthogonal) matching pursuit [Gilbert, Tropp]
iterated thresholding [Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; …]
CoSaMP [Needell and Tropp] 38

39. Iterated Thresholding
update signal estimate
prune signal estimate (best K-term approx)
update residual

40. Summary: CS Encoding: = random linear combinations of the entries of
Decoding: Recover from via optimization
sparse signal
measurements
nonzero
entries 40

41. CS Hallmarks Stable Asymmetrical (most processing at decoder)
acquisition/recovery process is numerically stable
Asymmetrical (most processing at decoder)
conventional: smart encoder, dumb decoder
CS: dumb encoder, smart decoder
Democratic
each measurement carries the same amount of information
robust to measurement loss and quantization
“digital fountain” property
Random measurements encrypted
Universal
same random projections / hardware can be used for any sparse signal class (generic) 41

42. Compressive Sensing In Action

43. Gerhard Richter 4096 Farben / 4096 Colours
cm X 254 cm Laquer on Canvas Catalogue Raisonné: 359
Museum Collection: Staatliche Kunstsammlungen Dresden (on loan)
Sales history: 11 May 2004 Christie's New York Post-War and Contemporary Art (Evening Sale), Lot 34 US$3,703,500   43

44. “Single-Pixel” CS Camera
scene
single photon detector
image reconstruction or processing
DMD
DMD
random
pattern on
DMD array
DMD is used in projectors
Multiply value of random pattern in mirror with value of signal (light intensity) in pixel
lens is focused onto the photodiode
w/ Kevin Kelly

45. “Single-Pixel” CS Camera
scene
single photon detector
image reconstruction or processing
DMD
DMD
random
pattern on
DMD array
DMD is used in projectors
Multiply value of random pattern in mirror with value of signal (light intensity) in pixel
lens is focused onto the photodiode …
Flip mirror array M times to acquire M measurements
Sparsity-based (linear programming) recovery

46. First Image Acquisition
target pixels
11000 measurements (16%)
1300 measurements
(2%)

48. single photon detector
Utility?
Fairchild
100Mpixel
CCD
single photon detector
DMD is used in projectors
Multiply value of random pattern in mirror with value of signal (light intensity) in pixel
lens is focused onto the photodiode
DMD
DMD

49. CS Low-Light Imaging with PMT
true color low-light imaging
256 x 256 image with 10:1 compression
[Nature Photonics, April 2007]

50. CS Infrared Camera
20% 5%

51. CS Hyperspectral Imager
spectrometer
hyperspectral data cube nm
1M space x wavelength voxels
200k random sums

52. CS MRI Lustig, Pauly, Donoho et al at Stanford
Design MRI sampling patterns (in frequency/k-space) to be close to random
Speed up acquisition by reducing number of samples required for a given image reconstruction quality

53. Multi-Slice Brain Imaging [M. Lustig]

54. 3DFT Angiography [M. Lustig]

55. CS In Action CS makes sense when measurements are expensive
Ultrawideband A/D converters [DARPA “Analog to Information” program]
Camera networks
sensing/compression/fusion
Radar, sonar, array processing
exploit spatial sparsity of targets
DNA microarrays
smaller, more agile arrays for bio-sensing [Milenkovic, Orlitsky, B]

56. Summary Compressive sensing
randomized dimensionality reduction
integrates sensing, compression, processing
exploits signal sparsity information
enables new sensing modalities, architectures, systems
relies on large-scale optimization
Why CS works: preserves information in signals with concise geometric structure
sparse signals | compressible signals | manifolds

57. Open Research Issues Links with information theory
new encoding matrix design via codes (LDPC, fountains)
new decoding algorithms (BP, etc.)
quantization and rate distortion theory
Links with machine learning
Johnson-Lindenstrauss, manifold embedding, RIP
Processing/inference on random projections
filtering, tracking, interference cancellation, …
Multi-signal CS
array processing, localization, sensor networks, …
CS hardware
ADCs, receivers, cameras, imagers, radars, …

58. background subtracted images
Beyond Sparsity
Sparse signal model captures simplistic primary structure
pixels:
background subtracted images
wavelets:
natural images
Gabor atoms:
chirps/tones 58

59. background subtracted images
Structured Sparsity
Sparse signal model captures simplistic primary structure
Modern compression/processing algorithms capture richer secondary coefficient structure
pixels:
background subtracted images
wavelets:
natural images
Gabor atoms:
chirps/tones 59

60. Wavelet Tree-Sparse Recovery
CoSaMP, (RMSE=1.12)
target signal
N=1024 M=80
Tree-sparse CoSaMP (RMSE=0.037)
L1-minimization (RMSE=0.751) 60

61. dsp.rice.edu/cs

63. vibrant interactive community connected innovative up-to-date efficient effective

64. create
share
use
freely
re-use openly
vibrant interactive community connected innovative up-to-date efficient effective

65. create
share
use
freely
re-use openly
vibrant interactive community connected innovative up-to-date efficient effective

66. create share use re-use freely openly why?
today’s textbooks/courses lock up educational ideas
closed formats
closed copyrights

67. create share
use re-use
freely openly
open education

68. OE enablers

69. enabler 1: technology Web/XML modularization
common framework for sharing
Internet
virtually free distribution
virtually infinite, permanent storage

70. Connexions – primordial state

71. textbook / course

72. personalize

73. reuse

74. enabler 2: new IP common legal vocabulary
intellectual property and copyright
make content safe to share
common legal vocabulary
inspiration: open-source software (ex: Linux)

75. today’s textbook pipeline
authoring
editing
quality control
publishing
distribution

76. open education ecosystem
authoring
editing
quality control
publishing
distribution
feedback
peers
users
learning

77. Connexions (cnx.org) free on-line low-cost in print
non-profit OE platform/community based at Rice U since 1999 ($6m+ in foundation funding)
950+ open textbooks/courses
reusable Lego modules
1.5 million+ unique users/month
free on-line
low-cost in print

79. Universality
Random measurements can be used for signals sparse in any basis

80. Universality
Random measurements can be used for signals sparse in any basis

81. Universality
Random measurements can be used for signals sparse in any basis
sparse coefficient
vector
nonzero
entries