Introduction to Compressed Sensing and its applications mehul.raval@ahduni.edu.in

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Age of Digital World Our life revolve around the digital world Entertainment, communication, business, life !! Digital bits streams running at the background is expected to deliver “natural” performance. Surround sound, 3D TV, sixth sense !! Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Human centric conversion process Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Nyquist Theorem Band limited signal with highest frequency of B Hz can be reconstructed perfectly from its samples with rate > 2B. (Nyquist Rate). Relation in X(t) and X(nT). Digital operation replacing analog counter parts. Relationship in power spectral densities of analog and discrete random process. Estimation and detection by DSP is possible. 21-May-14 Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Spectrums of time domain signal and its samples Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Wide band signal acquisition Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 ADC Analog to digital converters forms heart. Physical (analog) information streams of numbers  digital processing by software. Intriguing task Snap shot of fast varying signal + acquiring measurements. Unprecedented strain on ADC’s and DSP. Demand is ever increasing. 21-May-14 Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Ever increasing demand After sampling, we retain large number of bits. Conventional solution to storage space Sampling  Compression (exploiting redundancy) Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Ultimate question[1] Why so much effort is spent on acquiring (sampling) the data(redundancy) when most of it will be thrown away (compression)? Can’t we directly measure the part (information) which will not be thrown away? Why can’nt we ask such a question?? [1] D.L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no.4, pp. 1289-1306, Sep. 2006. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Idea of simultaneous compression and sampling Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Common link Basis Function Coefficients Signal k - sparse signal 𝑥 ≅ Φ𝜃 Support of non zero indices for 𝜃 denoted as 𝑆(𝜃) . Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Transform domain image representation Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Basics of Compressed Sensing Demo Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Whittaker–Shannon–Kotelnikov (WSK) v/s Compressive sensing Greater than Nyquist rate. Uniform / Non uniform sampling Non uniform sampling is based on Lagrange interpolation. Theory developed for continuous time signals CS Sub Nyquist sampling. Randomized measurement matrix. Samples are inner product. Initial focus on finite dimensional signal. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Whittaker–Shannon–Kotelnikov (WSK) v/s Compressive sensing No underlying signal structure. Transform coding based compression. Does not consider Hardware implementation CS Initial sparse. Now structure beyond sparsity is explored. Signal structure over and above sparsity gives higher compression. Structured non random measurement matrix. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Basic CS frame work . 𝑥 is a N x 1 vector, 𝑦 is a M x 1 vector with M << N is a random measurement matrix. Sparsifying dictionary of basis Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Recovery Recover Given: and : Class with all k sparse signal. CS makes exhaustive search in such that Solution using optimization problem Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Issues with optimization is computationally very expensive: it is an non-deterministic polynomial-time (NP) hard problem. E.g. M = 1000, N = 5000, k = 100 search space. Non convex Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Questions to be tackled 1. Can this problem be solved by any other mechanisms? 2. How can we get an estimated solution and what level of estimation accuracy is acceptable? (after all Engineers always looks for the workable approximate solutions !!!); 3. What kind of approximation will yield the solution closer to the desired one? Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Convex and Non convex set Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Convex relaxation - norm . Set is convex, so above problem is also convex. A strictly posed convex problems leads to a close form solution and guaranteed to converge at the local minima. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 norm Find Xi’s for which norm is minimal. It is not strictly convex and it may have multiple solution. However, these solutions are 1. clustered around in a convex set as all optimal solutions will have an penalty and their combination would also be convex; 2. the set is bounded; 3. among them at least one has at most k non zero elements. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Best possible approximation approximation ℓ𝑝 norm with 𝑝<1 . Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Heart: The measurement matrix ; : with M << N Basis Fixed independent of signal. Most fundamental design questions: 1. how much of information about signal x is retained in its linear measurements y ?; 2. how can the linear measurements y uniquely represent x?; 3. how can the original signal be recovered from its measurement? Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

The measurement matrix It is rank deficient with non empty Null space Consequence: Unable to recover the signal from measurements. Design: For distinct k sparse signals Should have a unique measurement Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Characterization of Uniqueness Spark = Sparse + Rank If spark( ) > 2k  uniquely represents a k- sparse signal belonging to class The spark of the measurement matrix is used to ensure stability and consistency. Computing is search over all sub-matrices. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Coherence Coherence can be used to identify the sparse signal. It is describing the dependency between two columns. Supremum on sparsity k is Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Uniqueness property in presence of noise Modified In CS, it is assumed that is available during sparse signal recovery. Measurement process should be robust to such noise. Sparse recovery is possible if Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Restricted Isometric property (RIP) A matrix satisfies (k, ) restricted iso-metry property of order k if for , Isometry is a function between the two spaces which has a property to preserve distance between each pair of points. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Building Sensing Matrices . Building Sensing Matrices . Vandermonde matrix has spark M + 1 : Geometric progression. Poor conditioning. Gabor Frame = n x n time shift matrix Bernoulli, Gaussian, or sub Gaussian Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Research avenue Structured measurement matrix: Application dependent. Subjected to the physical constraint of the application. M.F. Duarte & Y.C. Eldar, "Structured compressed sensing: from theory to applications," IEEE Trans. Sig. Proc., vol.59, no. 9, pp. 4053-4085, Sept. 2011. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 CS recovery CS recovery = L1 minimization The choice of algorithm is based on various factors, namely: 1. signal reconstruction timing from measurement vector; 2. the number of measurement required for recovery to determine the storage requirements; 3. the simplicity of the implementation; 4. possible portability to the hardware for execution; 5. fidelity of the signal recovery. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 BP: Basis Pursuit, BPIC: Basis Pursuit with Inequality Constraint, BPDN: Basis Pursuit De-noising, MP: Matching Pursuit, OMP: Orthogonal Matching Pursuit, IHT: Iterative Hard Thresholding 21-May-14 Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Basic Its power stems from the fact that it converts the search into convex problem and provides the accurate recovery. Basis pursuit (BP): norm has a tendency to locate the sparse solutions if ever they exist. 21-May-14 Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 2D reconstruction (a) (b) a) original Image N = 6400 (80 x 80); b) reconstructed Image with M = 2400, MSE: 0.0 reconstruction time: 18.64 sec. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 BPIC Bound based on noise. Could be user defined Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 N = 1024; k = 50; M = 220; MSE=2.02 x 10-4 Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 (a) (b) (a) reconstructed Image M = 1600, MSE: 0.25, reconstruction time: 16.27 sec; (b) reconstructed Image M = 800, MSE: 0.31, reconstruction time: 31.19 sec Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Watermarking application Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Detector Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Over determined system with M > N Construct matrix s.t = 0. Estimate using CS formulation. s.t Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Fidelity Non malicious Malicious manipulations Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Single Pixel Camera webee.technion.ac.il Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 Summary One of the most exciting domain. Interdisciplinary: signal processing, statistics, probability theory, computer science, optimization, linear programming. Look beyond the random measurement matrix. Developing a better signal models: finite rate innovation (FRI), Xampling framework Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 References D.L. Donoho, "Compressed sensing", IEEE Trans. Inf. Theory, vol. 52, no.4, pp. 1289 - 1306, Sep. 2006. E.J. Candès & T.Tao, "Near optimal signal recovery from random projections: Universal encoding strategies," IEEE Trans. Inf. Theory, vol.52, no. 12, pp.5406 - 5425, Dec. 2006. E.J. Candès, J. Romberg, & T. Tao, "Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489 - 509, Dec. 2006. Richard Baraniuk, "Compressive sensing," IEEE Sig. Proc. Mag., vol. 24, no. 4, pp. 118 -124, 2007. M.F. Duarte & Y.C. Eldar, "Structured compressed sensing: from theory to applications," IEEE Trans. Sig. Proc., vol.59, no. 9, pp. 4053-4085, Sept. 2011. Compressed sensing, Theory and applications, (eds. Y.C. Eldar & Gitta Kutyniok), Cambridge university press, Cambridge, UK, 2012. E. J. Candès, J. Romberg and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements”, Comm. Pure Appl. Math., vol.59,pp. 1207–1223, 2006. B. K. Natarajan, “Sparse approximate solutions to linear systems”, SIAM Journal on computing, vol. 24, pp.227–234, 1995. Nonlinear Optimization: Complexity Issues, S. A. Vavasis, Oxford University Press, New York, 1991. E. J. Candès and T. Tao, "The power of convex relaxation: Near-optimal matrix completion," IEEE Trans. Inform. Theory, vol. 56, no. 5, pp. 2053-2080, 2009. Convex Optimization, Stephen Boyd & Lieven Vandenberghe, Cambridge University Press, 2004. S. S. Chen, D. L. Donoho, & M. A. Saunders, " Atomic decomposition by basis pursuit," SIAM J. Scientific Computing, vol. 20, no. 1, pp.33–61, 1998. Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Compressive sensing solvers http://users.ece.gatech.edu/~justin/l1magic/ http://sparselab.stanford.edu/ http://www.lx.it.pt/~mtf/GPSR/ http://www.stanford.edu/~boyd/l1_ls/ http://www.personal.soton.ac.uk/tb1m08/sparsify/sparsify.html http://www.lx.it.pt/~mtf/SpaRSA/ https://sites.google.com/site/igorcarron2/cs#reconstruction (Comprehensive Listing of solvers) http://videolectures.net/bmvc09_chellappa_cscv/ Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015

Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015 mehul.raval@ahduni.edu.in Thank you Workshop on Compressive sensing, MNIT Jaipur, 12th - 13th Sept. 2015