“A fast method for Underdetermined Sparse Component Analysis (SCA) based on Iterative Detection- Estimation (IDE)” Arash Ali-Amini 1 Massoud BABAIE-ZADEH 1,2 Christian Jutten 2 1- Sharif University of Technology, Tehran, IRAN 2- Laboratory of Images and Signals (LIS), CNRS, INPG, UJF, Grenoble, FRANCE

MaxEnt2006, 8-13 July, Paris, France 2/42 Outline Introduction to Blind Source Separation Geometrical Interpretation Sparse Component Analysis (SCA), underdetermined case  Identifying mixing matrix  Source restoration Finding sparse solutions of an Underdetermined System of Linear Equations (USLE):  Minimum L0 norm  Matching Pursuit  Minimum L1 norm or Basis Pursuit (  Linear Programming)  Iterative Detection-Estimation (IDE) - our method Simulation results Conclusions and Perspectives

MaxEnt2006, 8-13 July, Paris, France 3/42 Blind Source Separation (BSS) Source signals s 1, s 2, …, s M Source vector: s=(s 1, s 2, …, s M ) T Observation vector: x=(x 1, x 2, …, x N ) T Mixing system  x = As Goal  Finding a separating matrix y = Bx

MaxEnt2006, 8-13 July, Paris, France 4/42 Blind Source Separation (cont.) Assumption:  N >=M (#sensors >= #sources)  A is full-rank (invertible) prior information: Statistical “Independence” of sources Main idea: Find “B” to obtain “independent” outputs (  Independent Component Analysis=ICA)

MaxEnt2006, 8-13 July, Paris, France 5/42 Blind Source Separation (cont.) Separability Theorem [Comon 1994,Darmois 1953]: If at most 1 source is Gaussian: statistical independence of outputs  source separation (  ICA: a method for BSS) Indeterminacies: permutation, scale A = [a 1, a 2, …, a M ], x=As  x = s 1 a 1 + s 2 a 2 + … + s M a M

MaxEnt2006, 8-13 July, Paris, France 6/42 Geometrical Interpretation x=As Statistical Independence of s1 and s2, bounded PDF  rectangular support for joint PDF of (s1,s2)  rectangular scatter plot for (s1,s2)

MaxEnt2006, 8-13 July, Paris, France 7/42 Sparse Sources s-planex-plane 2 sources, 2 sensors:

MaxEnt2006, 8-13 July, Paris, France 8/42 Importance of Sparse Source Separation The sources may be not sparse in time, but sparse in another domain (frequency, time- frequency, time-scale, etc.) x=As  T{x}=A T{s}  T: a ‘sparsifying’ transform Example. Speech signals are usually sparse in ‘time-frequency’ domain.

MaxEnt2006, 8-13 July, Paris, France 9/42 Sparse sources (cont.) 3 sparse sources, 2 sensors Sparsity  Source Separation, with more sensors than sources?

MaxEnt2006, 8-13 July, Paris, France 10/42 Estimating the mixing matrix A = [a 1, a 2, a 3 ]  x = s 1 a 1 + s 2 a 2 + s 3 a 3  Mixing matrix is easily identified for sparse sources Scale & Permutation indeterminacy ||a i ||=1

MaxEnt2006, 8-13 July, Paris, France 11/42 Restoration of the sources How to find the sources, after having found the mixing matrix (A)? 2 equations, 3 unknowns  infinitely many solutions! Underdertermined SCA, underdetermined system of equations

MaxEnt2006, 8-13 July, Paris, France 12/42 Identification vs Separation #Sources <= #Sensors: Identifying A  Source Separation #Sources > #Sensors (underdetermined) Identifying A  Source Separation  Two different problems: Identifying the mixing matrix (relatively easy) Restoring the sources (difficult)

MaxEnt2006, 8-13 July, Paris, France 13/42 Is it possible, at all? A is known, at eash instant (n 0 ), we should solve un underdetermined linear system of equations: Infinite number of solutions s(n 0 )  Is it possible to recover the sources?

MaxEnt2006, 8-13 July, Paris, France 14/42 ‘Sparse’ solution s i (n) sparse in time  The vector s(n 0 ) is most likely a ‘sparse vector’ A.s(n 0 ) = x(n 0 ) has infinitely many solutions, but not all of them are sparse! Idea: For restoring the sources, take the sparsest solution (most likely solution)

MaxEnt2006, 8-13 July, Paris, France 15/42 Example ( 2 equations, 4 unknowns ) Some of solutions: Sparsest

MaxEnt2006, 8-13 July, Paris, France 16/42 The idea of solving underdetermined SCA A s(n) = x(n), n=0,1,…,T Step 1 (identification): Estimate A (relatively easy) Step 2 (source restoration): At each instant n 0, find the sparsest solution of A s( n 0 ) = x( n 0 ), n 0 =0,…,T Main question: HOW to find the sparsest solution of an Underdetermined System of Linear Equations (USLE)?

MaxEnt2006, 8-13 July, Paris, France 17/42 Another application of USLE: Atomic decomposition over an overcompelete dictionary Decomposing a signal x, as a linear combination of a set of fixed signals (atoms) Terminology: Atoms:  i, i=1,…,M Dictionary: {  1,  2,…,  M }

MaxEnt2006, 8-13 July, Paris, France 18/42 Atomic decomposition (cont.) M=N  Complete dictionary  Unique set of coefficients Examples: Dirac dictionary, Fourier Dictionary Dirac Dictionary:

MaxEnt2006, 8-13 July, Paris, France 19/42 Atomic decomposition (cont.) M=N  Complete dictionary  Unique set of coefficients Examples: Dirac dictionary, Fourier Dictionary Fourier Dictionary:

MaxEnt2006, 8-13 July, Paris, France 20/42 Atomic decomposition (cont.) Matrix Form: If just a few number of coefficient are non-zero  The underlying structure is very well revealed Example.  signal has just a few non-zero samples in time  its decomposition over the Dirac dictionary reveals it  Signals composed of a few pure frequencies  its decomposition over the Fourier dictionary reveals it  How about a signals which is the sum of a pure frequency and a dirac?

MaxEnt2006, 8-13 July, Paris, France 21/42 Atomic decomposition (cont.) Solution: consider a larger dictionary, containing both Dirac and Fourier atoms M>N  Overcomplete dictionary. Problem: Non-uniqueness of  (Non-unique representation) (  USLE) However: we are looking for sparse solution

MaxEnt2006, 8-13 July, Paris, France 22/42 Sparse solution of USLE Underdetermined SCA Atomic Decomposition on over-complete dictionaries Findind sparsest solution of USLE

MaxEnt2006, 8-13 July, Paris, France 23/42 Uniqueness of sparse solution x=As, n equations, m unknowns, m>n Question: Is the sparse solution unique? Theorem (Donoho 2004): if there is a solution s with less than n/2 non-zero components, then it is unique with probability 1 (that is, for almost all A’s).

MaxEnt2006, 8-13 July, Paris, France 24/42 How to find the sparsest solution A.s = x, n equations, m unknowns, m>n Goal: Finding the sparsest solution Note: at least m-n sources are zero. Direct method:  Set m-n (arbitrary) sources equal to zero  Solve the remaining system of n equations and n unknowns  Do above for all possible choices, and take sparsest answer. Another name: Minimum L 0 norm method  L 0 norm of s = number of non-zero components =  |s i | 0

MaxEnt2006, 8-13 July, Paris, France 25/42 Example s1=s2=0  s=(0, 0, 1.5, 2.5) T  L 0 =2 s1=s3=0  s=(0, 2, 0, 0) T  L 0 =1 s1=s4=0  s=(0, 2, 0, 0) T  L 0 =1 s2=s3=0  s=(2, 0, 0, 2) T  L 0 =2 s2=s4=0  s=(10, 0, -6, 0) T  L 0 =2 s3=s4=0  s=(0, 2, 0, 0) T  L 0 =2  Minimum L 0 norm solution  s=(0, 2, 0, 0) T

MaxEnt2006, 8-13 July, Paris, France 26/42 Drawbacks of minimal norm L 0 Highly (unacceptably) sensitive to noise Need for a combinatorial search: Example. m=50, n=30, On our computer: Time for solving a 30 by 30 system of equation=2x10 -4 s Total time  (5x10 13 )(2x10 -4 )  300 years!  Non-tractable

MaxEnt2006, 8-13 July, Paris, France 27/42 faster methods? Matching Pursuit [Mallat & Zhang, 1993] Basis Pursuit (minimal L1 norm  Linear Programming) [Chen, Donoho, Saunders, 1995] Our method (IDE)

MaxEnt2006, 8-13 July, Paris, France 28/42 Matching Pursuit (MP) [Mallat & Zhang, 1993]

MaxEnt2006, 8-13 July, Paris, France 29/42 Properties of MP Advantage:  Very Fast Drawback  A very ‘greedy’ algorithm  Mistake in a stage can never be corrected  Not necessarily a sparse solution x=s1a1+s2a2x=s1a1+s2a2 a1 a2 a3 a5 a4

MaxEnt2006, 8-13 July, Paris, France 30/42 Minimum L 1 norm or Basis Pursuit [Chen, Donoho, Saunders, 1995] Minimum norm L1 solution: MAP estimator under a Laplacian prior Recent theoretical support (Donoho, 2004) : For ‘most’ ‘large’ underdetermined systems of linear equations, the minimal L 1 norm solution is also the sparsest solution

MaxEnt2006, 8-13 July, Paris, France 31/42 Minimal L 1 norm (cont.) Minimal L 1 norm solution may be found by Linear Programming (LP) Fast algorithms for LP:  Simplex  Interior Point method

MaxEnt2006, 8-13 July, Paris, France 32/42 Minimal L 1 norm (cont.) Advantages:  Very good practical results  Theoretical support Drawback:  Tractable, but still very time-consuming

MaxEnt2006, 8-13 July, Paris, France 33/42 Iterative Detection-Estmation (IDE)- Our method Main Idea:  Step 1 (Detection): Detect which sources are ‘active’, and which are ‘non-active’  Step 2 (Estimation): Knowing active sources, estimate their values Problem: Detecting the activity status of a source, requires the values of all other sources! Our proposition: Iterative Detection-Estimation Activity Detection  Value Estimation

MaxEnt2006, 8-13 July, Paris, France 34/42 IDE, Detection step Assumptions:  Mixture of Gaussian model for sparse sources: s i active  s i ~N(0,  1 2 ) s i in-active  s i ~N(0,  0 2 ),  1 >>  0  0 : probability of in-activity  1  Columns of A are normalized Hypotheses: Proposition:  t=a 1 T x=s 1 +   Activity detection test: g 1 (s)=|t-  |>    depends on other sources  use their previous estimates

MaxEnt2006, 8-13 July, Paris, France 35/42 IDE, Estimation step Estimation equation: Solution using Lagrange multipliers: Closed-form solution  See the paper 

MaxEnt2006, 8-13 July, Paris, France 36/42 IDE, summary Detection Step (resulted from binary hypothesis testing, with a Mixture of Gaussian source model): Estimation Step:

MaxEnt2006, 8-13 July, Paris, France 37/42 IDE, simulation results m=1024, n=0.4m=409

MaxEnt2006, 8-13 July, Paris, France 38/42 IDE, simulation results (cont.) m=1024, n=0.4m=409 IDE is about two order of magnitudes faster than LP method (with interior point optimization).

MaxEnt2006, 8-13 July, Paris, France 39/42 Speed/Complexity comparision

MaxEnt2006, 8-13 July, Paris, France 40/42 IDE, simulation results: number of possible active sources m=500, n=0.4m=200, averaged on 10 simulations

MaxEnt2006, 8-13 July, Paris, France 41/42 Conclusion and Perspectives Two problems of Underdetermined SCA:  Identifying mixing matrix  Restoring sources Two applications of finding sparse solution of USLE’s:  Source restoration in underdetermined SCA  Atomic Decomposition on over-complete dictionaries Methods:  Minimum L0 norm (  Combinatorial search)  Minimum L1 norm or Basis Pursuit (  Linear Programming)  Matching Pursuit  Iterative Detection-Estimation (IDE) Perspectives:  Better activity detection (removing thresholds?)  Applications in other domains

MaxEnt2006, 8-13 July, Paris, France 42/42 Thank you very much for your attention