1. IEEE UKRI Talk, De Montfort Univ., April 2, 04 1 Open Issues in Constrained Blind Source Separation Jonathon Chambers Cardiff Professorial Research Fellow Cardiff School of Engineering Cardiff University, Wales, U.K. E-mail: chambersj@cf.ac.uk

2. IEEE UKRI Talk, De Montfort Univ., April 2, 04 2 Summary of Talk Acknowledgement Historical background & motivation BSS with matrix constraints Penalty functions in FD-BSS Exploiting periodicity in BSS Future application-driven challenges

3. IEEE UKRI Talk, De Montfort Univ., April 2, 04 3 Acknowledgements Jonathon Chambers wishes to express his sincere thanks for the support of Professor Andrzej Cichocki, Riken Brain Science Institute, Japan The invitation from the organising committee of the workshop to give this talk. His co-researchers: Drs Saeid Sanei, Maria Jafari and Wenwu Wang.

4. IEEE UKRI Talk, De Montfort Univ., April 2, 04 4 LMS Algorithm B. Widrow, and M.E. Hoff, Jr., Adaptive switching circuits, IRE Wescon Conv. Rec., pt. 4, pp. 96-104, 1960. LMS Update

5. IEEE UKRI Talk, De Montfort Univ., April 2, 04 5 Historical Background The field of conventional adaptive signal processing has been greatly enhanced by the exploitation of constrained optimisation Constraints on the error, and/or structure or some norm of the weights via, for example, Lagrange multipliers and/or Karush-Khun-Tucker conditions

6. IEEE UKRI Talk, De Montfort Univ., April 2, 04 6 Historical Background Certain key papers: O.L. Frost, III, An algorithm for linearly constrained adaptive array processing, Proc. IEEE, Vol. 60(8), pp. 926-925, 1972 R.P. Gitlin et al. The tap-leakage algorithm: an algorithm for the stable operation of a digitally implemented fractionally spaced equalizer, Bell Sys. Tech. Journal, Vol. 61(8), pp. 1817-1839, 1982. D.T.M. Slock, Convergence behavior of the LMS and Normalised LMS Algorithms, IEEE Trans. Signal Processing, Vol. 41(9), pp. 2811-2825, 1993.

7. IEEE UKRI Talk, De Montfort Univ., April 2, 04 7 Historical Background Cont. S.C. Douglas, A family of normalized LMS algorithms, IEEE Signal Processing Letters, Vol. 1(3), pp. 49-51, 1994. S.C. Douglas, and M. Rupp, A posteriori updates for adaptive filters, Asilomar Conference on Signals, Systems and Computers, Vol. 2, pp 1641-1645, 1997. T. Gänsler, et al., A robust proportionate affine projection algorithm for network echo cancellation, Proc. ICASSP 2000, Vol. 2, pp. 793-796, 2000. O. Vainia, Polynomial constrained LMS adaptive algorithm for measurement signal processing, Proc. IECON 2002, Vol. 2, pp. 1479-1482, 2002.

8. IEEE UKRI Talk, De Montfort Univ., April 2, 04 8 Motivation In many applications of Independent Component Analysis (ICA) and Blind Source Separation (BSS) estimated source signals and the mixing or separating matrices have some special structure or some constraints are imposed for the matrices…, Cichocki and Georgiev, 2003

9. IEEE UKRI Talk, De Montfort Univ., April 2, 04 9 H s1s1 sNsN W x1x1 xMxM Y1Y1 YNYN UnknownKnown Independent? Adapt Mixing ProcessUnmixing Process Fundamental Model for Instantaneous Blind Source Separation

10. IEEE UKRI Talk, De Montfort Univ., April 2, 04 10 Certain BSS Books Andrzej Cichocki and Shun-Ichi Amari, Adaptive Blind Signal and Image Processing, Wiley, 2002 Simon Haykin Unsupervised Adaptive Filtering, Vols. I and II, Wiley, 2000 Aapo Hyvärinen, Juha Karhunen and Erkki Oja, Independent Component Analysis, Wiley, 2001 Te-Won Lee, Independent component analysis: theory and applications, Kluwer, 1998

11. IEEE UKRI Talk, De Montfort Univ., April 2, 04 11 BSS References A. Mansour and M. Kawamoto, ICA Papers Classified According to their Applications and Performances, IEICE Trans. Fundamentals, Vol. E86-A, No. 3, March 2003, pp. 620-633. In 2002, 800 different papers have been published, these are downloadable at http://ali.mansour.free/REF.htm http://ali.mansour.free/REF.htm

12. IEEE UKRI Talk, De Montfort Univ., April 2, 04 12 With a symmetric mixing matrix [C&G,2003]:- BSS With Matrix Constraints

13. IEEE UKRI Talk, De Montfort Univ., April 2, 04 13 Stable Frobenius norm of the separating matrix Theorem [C&G 2003]: The learning rule where β > 0 is a scaling factor and γ(t) = trace(W T (t)F(y(t))W(t)) > 0, stabilizes the Frobenius norm of W(t) such that BSS With Matrix Consts. Cont.

14. IEEE UKRI Talk, De Montfort Univ., April 2, 04 14 Consequence: The modified NG descent learning algorithm, with a forgetting factor, described as with γ(t) = -trace(W T (t)[ J(W)/ W]W T (t)W(t)) > 0 has a W(t) with bounded Frobenius norm throughout the learning process. BSS With Matrix Consts. Cont.

15. IEEE UKRI Talk, De Montfort Univ., April 2, 04 15 Prof. Amaris Leaky NG Algorithm becomes where 0 << (1-βγ(t)η(t)) < 1 is the leakage factor BSS With Matrix Consts. Cont.

16. IEEE UKRI Talk, De Montfort Univ., April 2, 04 16 Introducing a semi-orthogonality constraint so that it is possible to extract an arbitrary group of sources, say e, 1 e N. Assuming pre-whitened data and the mixing matrix A = QH, the demixing matrix W e should satisfy W e A = [I e,0 N-e ] BSS With Matrix Consts. Cont.

17. IEEE UKRI Talk, De Montfort Univ., April 2, 04 17 A natural gradient algorithm to find W e becomes:- With initial conditions which satisfy BSS With Matrix Consts. Cont.

18. IEEE UKRI Talk, De Montfort Univ., April 2, 04 18 Real Convolutive Mixing Env. – Cocktail Party Problem

19. IEEE UKRI Talk, De Montfort Univ., April 2, 04 19 Convolution Compact form: Expansion form: Convolutive BSS – Model

20. IEEE UKRI Talk, De Montfort Univ., April 2, 04 20 Taxonomy of Existing Sols. To Convolutive BSS Performing blind separation in the time domain by extending the existing instantaneous methods to conv. case Transforming the convolutive BSS problem into multiple instantaneous (complex) problems in the frequency domain Decomposing the system into smaller problems using, for example, a subband approach Hybrid frequency and time domain approaches

21. IEEE UKRI Talk, De Montfort Univ., April 2, 04 21 DFT Convolutive BSS problem Multiple complex-valued instantaneous BSS problems Transform Convolutive BSS into the Frequency Domain

22. IEEE UKRI Talk, De Montfort Univ., April 2, 04 22 In the frequency domain:- Mathematical Formulation

23. IEEE UKRI Talk, De Montfort Univ., April 2, 04 23 De-mixing Operation

24. IEEE UKRI Talk, De Montfort Univ., April 2, 04 24 Constrained Optimisation and Joint Diagonalisation

25. IEEE UKRI Talk, De Montfort Univ., April 2, 04 25 Joint Diagonalisation Criterion Exploiting the non-stationarity of speech signals measured by the cross-spectrum of the output signals,

26. IEEE UKRI Talk, De Montfort Univ., April 2, 04 26 Exterior Penalty Function Approach

27. IEEE UKRI Talk, De Montfort Univ., April 2, 04 27 Exterior Penalty Function Approach Typical exterior penalty functions, and the shadow area represents the feasible set.

28. IEEE UKRI Talk, De Montfort Univ., April 2, 04 28 Proposed General Cost Function With a factor vector κ to incorporate exterior penalty functions, our cost function becomes:-

29. IEEE UKRI Talk, De Montfort Univ., April 2, 04 29 Numerical Experiments Use an exterior penalty function Employ a variant of gradient adaptation Utilize the filter length constraint to address the permutation problem (Parra & Spence) System with two inputs and two outputs (TITO!) H(z) = [{1 1.9 -0.75}, z -5 {0.5 0.3 0.2}; z -5 {-0.7 -0.3 -0.2}, {0.8 -0.1}]; D = 7, T = 1024, K = 5.

30. IEEE UKRI Talk, De Montfort Univ., April 2, 04 30 Convergence Performance of the New Criterion as a function of κ

31. IEEE UKRI Talk, De Montfort Univ., April 2, 04 31 Room Environment Experiment Use roommix function due to Westner Room 10x10x10m 3 cube Wall reflections calculated up to fifth order, atten. factor 0.5 SIR is plotted as a function of length of the separating system

32. IEEE UKRI Talk, De Montfort Univ., April 2, 04 32 Room Environment

33. IEEE UKRI Talk, De Montfort Univ., April 2, 04 33 Room Environment SIR

34. IEEE UKRI Talk, De Montfort Univ., April 2, 04 34 S1S1 S2S2 x1x1 x2x2 FDICA S 1 ×0.5 S 2 ×1 S 2 ×0.3 S 1 ×1.2 S 2 ×0.6 S 1 ×0.4 Permutation Problem in FD-CBSS

35. IEEE UKRI Talk, De Montfort Univ., April 2, 04 35 Summary of Existing Solutions to Permut. Problem in FD-CBSS Constraints on the filter models in the frequency domain Using special structure contained in signals Merging beamforming view to align solutions Exploiting the continuity of the spectra of the recovered signals – could coupled hidden Markov Models be used? What happens when the sources move, enter/re-enter the environment? What is the way forward?

36. IEEE UKRI Talk, De Montfort Univ., April 2, 04 36 Exploiting Source (Pseudo) - Periodicity W. Wang, M.G. Jafari, S. Sanei, and J.A. Chambers, Blind source separation of convolutive mixtures of cyclostationarity, to appear in the Special Issue on BSS, International Journal of Adaptive Control and Signal Processing, Guest Editor: Mike Davies, Queen Marys College, University of London H. Swada, R. Mukai, S. Araki, and S. Makino, A robust and precise method for solving the permutation problem of frequency-domain blind source separation, ICA 2003, Nara, Japan, 2003, pp. 505-510.

37. IEEE UKRI Talk, De Montfort Univ., April 2, 04 37 A natural gradient update exploiting cyclostationarity The Cyclostationary NGA uses the update equation where and p is the cycle frequency of the p-th source

38. IEEE UKRI Talk, De Montfort Univ., April 2, 04 38 A natural gradient update exploiting periodicity The Periodic NGA type update equation where

39. IEEE UKRI Talk, De Montfort Univ., April 2, 04 39 Emerging Applications Biomedical:- ECG, EEG, MEG and their integration Microarray time courses Measurements from the nano-lab http://www.nmrc.ie/research/transducers-group/trends.html http://www.nanospace.systems.org/ns_2000/NS00_Sessions.htm http://nanomed.ncl.ac.uk/m2l.htm Star Trek: The Tri-corder

40. IEEE UKRI Talk, De Montfort Univ., April 2, 04 40 Emerging Applications T. Bowles, J. Chambers, and A. Jakobsson, Advanced spectral estimation for the identification of cell-cycle regulated genes, IEEE EMBS UK and RI Postgraduate Conf in Biomedical Engineering and Medical Physics, 2003. X. Liao, and L. Carin, Constrained independent component analysis of DNA microarray signals, IEEE Workshop on Genomic Signal Processing and Statistics, 2002. S-I, Lee, and S. Batzoglou, Discovering biological processes from microarray data using independent component analysis, Dept EE/CS, Stanford Univ.

41. IEEE UKRI Talk, De Montfort Univ., April 2, 04 41 Summary The exploitation of constrained optimisation has been fundamental to the development and application of adaptive signal processing; this process is, however, very much in its infancy in blind source separation (BSS). Utilisation of certain a priori knowledge on the mixing matrices and the properties of the sources is likely to yield solutions to real-life SP problems. As such, the challenge for DSP engineers in the 21 st Century, is to advance the application of BSS methods in line with methods from adaptive signal processing.

42. IEEE UKRI Talk, De Montfort Univ., April 2, 04 42 Other References A. Cichocki, and P. Georgiev, Blind source separation algorithms with matrix constraints, IEICE Trans. Fundamentals, Vol. E86-A(3), March 2003, pp. 522-531. J.G. McWhirter, Mathematics and signal processing, Mathematics Today, April 2003, pp 47-54. W. Wang, S. Sanei, and J. Chambers, Penalty function based joint diagonalization approach for convolutive blind source separation, submitted to IEEE T-SP, Sept 2003.

43. IEEE UKRI Talk, De Montfort Univ., April 2, 04 43 Close ??? Mark Twain A man who swings a cat by its tail learns things he can learn no other way