ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Reconstruction Algorithms for Compressive Sensing II Presenter: 黃乃珊 Advisor: 吳安宇 教授 Date: 2014/04/08

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Schedule 2  EEII-225 日期 內容 Lab & HWSpeaker 3/11 Introduction to Compressive Sensing SystemNhuang 3/25 Reconstruction AlgorithmNhuang 4/8 Reconstruction AlgorithmLab1Nhuang 4/15 Break; 決定期末題目方向 4/22 Sampling Algorithm: Yumin 4/29 Midterm Presentation (Tutorial, Survey) 5/6 Application: Single Pixel CameraLab2Yumin 5/13 ~ 6/10 期末報告討論 6/24 Final Presentation

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Outline  Reconstruction Algorithms for Compressive Sensing  Bayesian Compressive Sensing  Iterative Thresholding  Approximate Message Passing  Implementation of Reconstruction Algorithms  Lab1: OMP Simulation  Reference 3

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Recovery Algorithms for Compressive Sensing  Linear Programming  Basis Pursuit (BP)  Greedy Algorithm  Matching Pursuit  Orthogonal Matching Pursuit (OMP)  Stagewise Orthogonal Matching Pursuit (StOMP)  Compressive Sampling Matching Pursuit (CoSaMP)  Subspace Pursuit (SP)  Iterative Thresholding  Iterative Hard Thresholding (IHT)  Iterative Soft Thresholding (IST)  Bayesian Compressive Sensing (BCS)  Approximate Matching Pursuit (AMP) 4

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Compressive Sensing in Mathematics 5 Sampling Reconstruction Channel

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Compressive Sensing in Linear Algebra  Reconstruction is composed of two parts:  Localize nonzero terms  Approximate nonzero value  Do correlation to find the location of non-zero terms  Solve least square problem to find the value  Projection (pseudo-inverse) 6 coefficient basis = MeasurementInput

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Orthogonal Matching Pursuit (OMP) [3]  Use greedy algorithm to iteratively recover sparse signal  Procedure: 1.Initialize 2.Find the column that is most correlated 3.Set Union (add one col. every iter.) 4.Solve the least squares 5.Update data and residual 6.Back to step 2 or output 7 [14]

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Iterative Threshold [4]  Iterative hard thresholding (IHT)  Iterative soft thresholding (IST) [2] 8

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Compressive Sensing From Mathematics to Engineering  Fourier transform was invented in 1812, and published in Not until FFT was developed in 1965, Fourier transform started to change the world.  Hardware design is limited by algorithm  Engineering perspective can help compressive sensing more powerful in practical application 9

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Message Passing  Messages pass from sender to receiver  Reliable transfer, and deliver in order  Belief propagation (BP)  Sum-product message passing  Calculate distribution for unobserved nodes on graph  Ex. low-density parity-check codes (LDPC), turbo codes  Approximate message passing (AMP) [8][9][10] 10

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Approximate Message Passing (AMP)  Iterative soft thresholding (IST)  Approximate message passing (AMP) [8][9][10]  Onsager reaction term cancels the self-feedback effects  Approximate sum-product messages for basis pursuit  Fast and good performance, but not suit for all random input 11

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Relevance Vector Machine (RVM)  Use Bayesian inference for regression and probabilistic classification  Support Vector Machine (SVM)  Classification and regression analysis  RVM is faster but at risk of local minima 12

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Bayesian Compressive Sensing [5][6][7]  Consider CS from Bayesian perspective  Provide a full posterior density function  Adopt the relevance vector machine (RVM)  Solve the problem of maximum a posterior (MAP) efficiently  Adaptive Compressive Sensing  Adaptively select projection with the goal to reduce uncertainty  Bayesian Compressive Sensing via Belief Propagation 13

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Compressive Sensing in Engineering A.Message passing  Sum-product message passing  Ex. Low-density parity-check codes (LDPC) B.Bayesian model  Bayesian learning, a kind of machine learning C.Adaptive filtering framework  Self-adjust to optimize desired signal 14 A. Message Passing B. Bayesian Model C. Adaptive Filter

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Outline  Reconstruction Algorithms for Compressive Sensing  Bayesian Compressive Sensing  Iterative Thresholding  Approximate Message Passing  Implementation of Reconstruction Algorithms  Lab1: OMP Simulation  Reference 15

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU  Choose Greedy rather than Linear programing  Optimization is better in terms of accuracy, but its implementation is very complex and time consuming.  Design issues  Matrix multiplication  Matrix inverse  Related works  OMP – ASIC & FPGA  CoSaMP – FPGA  IHT – GPU  AMP – ASIC & FPGA Implementation of Reconstruction Algorithms 16 Matrix Multiplication Matrix Inverse Processing Flow in Greedy Pursuits

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU OMP with Cholesky Decomposition  [11] is the earliest hardware implementation  Cholesky decomposition does not require square root calculations  Bottleneck  Kernel 1: 655/1645 cycles  Kernel 2 (Matrix inversion): 769/1645 cycles 17 (N, M, K)SQNRMax Freq.Latency OMP [11] ISCAS, 2010 (128,32,5)X39MHz24us OMP [13] ISSPA, 2012 (128,32,5)47dB107MHz16us [9] 1 2 3

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU OMP with QR Decomposition  Cholesky increases the latency with increasing dimension  QRD-RLS and fast inverse square algorithm are used in [14]  Remove columns with low coherence by an empirical threshold to reduce computational time  Tradeoff between MSE and reconstruction cycles 18 Reconstruction TimeNormalized MSE

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Outline  Reconstruction Algorithms for Compressive Sensing  Bayesian Compressive Sensing  Iterative Thresholding  Approximate Message Passing  Implementation of Reconstruction Algorithms  Lab1: OMP Simulation  Reference 19

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU OMP Simulation  Please design SolveOMP.m  Test the recovery performance of OMP with different size of measurement or different sparsity 20

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Reference [1] E. J. Candes, and M. B. Wakin, "An Introduction To Compressive Sampling," Signal Processing Magazine, IEEE, vol.25, no.2, pp.21-30, March 2008 [2] G. Pope, “Compressive Sensing – A Summary of Reconstruction Algorithm”, Swiss Federal Instituute of Technology Zurich [3] J. A. Tropp, A. C. Gilbert, “Signal Recovery from Random Measurements via Orthogonal Matching Pursuit,” IEEE Transactions on Information Theory, vol.53, no.12, pp , Dec [4] T. Blumensath, and M. E. Davies, "Iterative hard thresholding for compressed sensing." Applied and Computational Harmonic Analysis 27.3 (2009): [5] S. Ji, Y. Xue, and L. Carin, “Bayesian compressive sensing,” IEEE Trans. Signal Process., vol. 56, no. 6, pp. 2346–2356, Jun [6] M. E. Tipping, "Sparse Bayesian learning and the relevance vector machine." The Journal of Machine Learning Research 1 (2001): [7] D. Baron, S. Sarvotham, and R. G. Baraniuk, "Bayesian compressive sensing via belief propagation." Signal Processing, IEEE Transactions on 58.1 (2010): [8] D. L. Donoho, A. Maleki, and A. Montanari, "Message-passing algorithms for compressed sensing." Proceedings of the National Academy of Sciences (2009) [9] D. L. Donoho, A. Maleki, and A. Montanari, "Message passing algorithms for compressed sensing: I. motivation and construction." Information Theory Workshop (ITW), 2010 IEEE, Jan [10] D. L. Donoho, A. Maleki, and A. Montanari, "Message passing algorithms for compressed sensing: II. analysis and validation," Information Theory Workshop (ITW), 2010 IEEE, Jan

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Reference [11] A. Septimus, and R. Steinberg, "Compressive sampling hardware reconstruction," Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, vol., no., pp.3316,3319, May June [12] Lin Bai, P. Maechler, M. Muehlberghuber,and H. Kaeslin, "High-speed compressed sensing reconstruction on FPGA using OMP and AMP," Electronics, Circuits and Systems (ICECS), th IEEE International Conference on, vol., no., pp.53,56, 9-12 Dec [13] P. Blache, H. Rabah, and A. Amira, "High level prototyping and FPGA implementation of the orthogonal matching pursuit algorithm," Information Science, Signal Processing and their Applications (ISSPA), th International Conference on, vol., no., pp.1336,1340, 2-5 July 2012 [14] J.L.V.M. Stanislaus, and T. Mohsenin, "Low-complexity FPGA implementation of compressive sensing reconstruction," Computing, Networking and Communications (ICNC), 2013 International Conference on, vol., no., pp.671,675, Jan. 2013s 22