Outline Sparse Reconstruction RIP Condition Sparse Reconstruction Algorithms

Linear Equations with Noise Equations and Noise Equations: y = Ax + n Size of A: typically large Efficient Solution Using Gaussian elimination Using other numerical methods More Knowledge on Equation Vector x is sparse, very few nonzero elements of x Can utilize such structure for an efficient solution?

Sparsity in the Real World Sparsity in the original domain Radar signal, the sparse object reflection: sparsity in the time Signal detection from sparse directions Sparsity in the transform domain DCT coefficients of real images: negligible coefficients in the higher frequency components Narrow band interference, for the OFDM, in the frequency domain …

Sparse Reconstruction How to Model Sparsity Zero norm: ||x||0 Sparse vector x: small ||x||0 Sparse Reconstruction Problem Formulation Minimize the 0-norm under measurement constraints Noiseless: min ||x||0 s.t. Ax = b Noisy: min ||x||0 s.t. ||Ax – b||2 < ε Solve the 0-Norm Minimization Problem

Sparse Reconstruction Definition: Support of x - number of nonzero elements ||x||0 = |supp(x)| = |{i|xi ≠ 0}| x, K-sparse: ||x||0 ≤ K L0 Norm Highly Nonconvex L0 Norm Optimization Highly Non-convex Transform to a Convex Optimization Problem

L1 Norm Relaxation Transform Objective Function ||x||0 to ||x||1 L1 Norm Minimization Problem Noiseless: min ||x||1 s.t. Ax = b Noisy: min ||x||1 s.t. ||Ax – b||2 < ε L1 Norm: Convex Optimization Numerous Non-differential Points High computation complexity using brute-force L1- norm convex solution

L2 Norm Relaxation Transform Objective Function ||x||0 to ||x||2 Transformed L2 Problem Formulations min ||Ax – b||2, s.t. ||x||1 < q min ||Ax – b||2 + λ||x||1: the sparsity grows with the value of λ Advantage and Disadvantage Advantage: easy to optimize, differentiable at all points if no L1 terms Disadvantage: a lot of small non-zero elements, ||x||0 not really minimized

Restricted Isometry Property (RIP) Conditions on L1 Equivalence to L0 RIP condition The orthogonality of different columns of matrix A RIP Condition of Order K (1 - δ)||x||2 ≤ ||Ax|| ≤ (1 + δ)||x||2, for ||x||0 ≤ K completely orthogonal: δ = 0; The Non-orthogonality of Matrix A δK = inf{δ|(1 - δ)||x||2 ≤ ||Ax|| ≤ (1 + δ)||x||2, for all x in RK} completely orthogonal: δK = 0;

Restricted Isometry Property (RIP) Available Sufficient conditions on the L1 equivalence to L0: One of the following three conditions is satisfied Condition 1: δK + δ2K + δ3K < 1 Condition 2: δ2K < sqrt(2) – 1 Condition 3: δK < 0.307 Intuition of RIP: for completely δK = 0 Unitary matrix A What happened for unitary matrix A? L0, L1, …

Heuristic Solution to L1 Minimization Greedy Algorithm Successively finding the element that maximizes the correlation between basis and measurements “maximizes the correlation”? Correlation Maximization Maximize the correlation <rn, an>, residue rn Update the residue directly update: rn+1 = rn - <rn, an>an projection-based update: rn+1 = y – Anx, x = (AnHAn)-1AnHy

Heuristic Solution Matching Pursuit Orthogonal Matching Pursuit Each iteration, the residue: rn = b – Ax The column an, maximizing <rn, an>, is selected Update rn+1 = rn - <rn, an>an Orthogonal Matching Pursuit An = [a1a2…an], x = (AnHAn)-1AnHy Update the residue: rn+1 = y - Anx