Presentation is loading. Please wait.

Presentation is loading. Please wait.

Informatics and Mathematical Modelling / Intelligent Signal Processing 1 Morten Mørup Extensions of Non-negative Matrix Factorization to Higher Order data.

Similar presentations


Presentation on theme: "Informatics and Mathematical Modelling / Intelligent Signal Processing 1 Morten Mørup Extensions of Non-negative Matrix Factorization to Higher Order data."— Presentation transcript:

1 Informatics and Mathematical Modelling / Intelligent Signal Processing 1 Morten Mørup Extensions of Non-negative Matrix Factorization to Higher Order data Morten Mørup Informatics and Mathematical Modeling Intelligent Signal Processing Technical University of Denmark

2 Informatics and Mathematical Modelling / Intelligent Signal Processing 2 Morten Mørup Parts of the work done in collaboration with Sæby, May 22-2006 Lars Kai Hansen, Professor Department of Signal Processing Informatics and Mathematical Modeling, Technical University of Denmark Mikkel N. Schmidt, Stud. PhD Department of Signal Processing Informatics and Mathematical Modeling, Technical University of Denmark Sidse M. Arnfred, Dr. Med. PhD Cognitive Research Unit Hvidovre Hospital University Hospital of Copenhagen

3 Informatics and Mathematical Modelling / Intelligent Signal Processing 3 Morten Mørup Outline Non-negativity Matrix Factorization (NMF) Sparse coding (SNMF) Convolutive PARAFAC models (cPARAFAC) Higher Order Non-negative Matrix Factorization (an extension of NMF to the Tucker model)

4 Informatics and Mathematical Modelling / Intelligent Signal Processing 4 Morten Mørup NMF is based on Gradient Descent NMF: VWH s.t. W i,d,H d,j 0 Let C be a given cost function, then update the parameters according to:

5 Informatics and Mathematical Modelling / Intelligent Signal Processing 5 Morten Mørup The idea behind multiplicative updates Positive term Negative term

6 Informatics and Mathematical Modelling / Intelligent Signal Processing 6 Morten Mørup Non-negative matrix factorization (NMF) (Lee & Seung - 2001) NMF gives Part based representation (Lee & Seung – Nature 1999)

7 Informatics and Mathematical Modelling / Intelligent Signal Processing 7 Morten Mørup The NMF decomposition is not unique Simplical Cone NMF only unique when data adequately spans the positive orthant (Donoho & Stodden - 2004)

8 Informatics and Mathematical Modelling / Intelligent Signal Processing 8 Morten Mørup Sparse Coding NMF (SNMF) (Eggert & Körner, 2004)

9 Informatics and Mathematical Modelling / Intelligent Signal Processing 9 Morten Mørup Illustration (the swimmer problem) True Expressions Swimmer Articulations NMF Expressions SNMF Expressions

10 Informatics and Mathematical Modelling / Intelligent Signal Processing 10 Morten Mørup Why sparseness? Ensures uniqueness Eases interpretability (sparse representation  factor effects pertain to fewer dimensions) Can work as model selection (Sparseness can turn off excess factors by letting them become zero) Resolves over complete representations (when model has many more free variables than data points)

11 Informatics and Mathematical Modelling / Intelligent Signal Processing 11 Morten Mørup PART I: Convolutive PARAFAC (cPARAFAC)

12 Informatics and Mathematical Modelling / Intelligent Signal Processing 12 Morten Mørup By cPARAFAC means PARAFAC convolutive in at least one modality Convolution can be in any combination of modalities -Single convolutive, double convolutive etc. Convolution: The process of generating X by convolving (sending) the sources S through the filter A Deconvolution: The process of estimating the filter A from X and S

13 Informatics and Mathematical Modelling / Intelligent Signal Processing 13 Morten Mørup Relation to other models PARAFAC2 (Harshman, Kiers, Bro) Shifted PARAFAC (Hong and Harshman, 2003) cPARAFAC can account for echo effectscPARAFAC becomes shifted PARAFAC when convolutive filter is sparse 3 3

14 Informatics and Mathematical Modelling / Intelligent Signal Processing 14 Morten Mørup Application example of cPARAFAC Transcription and separation of music

15 Informatics and Mathematical Modelling / Intelligent Signal Processing 15 Morten Mørup The ‘ideal’ Log-frequency Magnitude Spectrogram of an instrument Different notes played by an instrument corresponds on a logarithmic frequency scale to a translation of the same harmonic structure of a fixed temporal pattern Tchaikovsky: Violin Concert in D Major Mozart Sonate no,. 16 in C Major

16 Informatics and Mathematical Modelling / Intelligent Signal Processing 16 Morten Mørup NMF 2D deconvolution (NMF2D 1 ): The Basic Idea Model a log-spectrogram of polyphonic music by an extended type of non-negative matrix factorization: –The frequency signature of a specific note played by an instrument has a fixed temporal pattern (echo)  model convolutive in time –Different notes of same instrument has same time-log- frequency signature but varying in fundamental frequency (shift)  model convolutive in the log-frequency axis. ( 1 Mørup & Scmidt, 2006)

17 Informatics and Mathematical Modelling / Intelligent Signal Processing 17 Morten Mørup NMF2D Model NMF2D Model – extension of NMFD 1 : ( 1 Smaragdis, 2004, Eggert et al. 2004, Fitzgerald et al. 2005)

18 Informatics and Mathematical Modelling / Intelligent Signal Processing 18 Morten Mørup Understanding the NMF2D Model

19 Informatics and Mathematical Modelling / Intelligent Signal Processing 19 Morten Mørup The NMF2D has inherent ambiguity between the structure in W and H To resolve this ambiguity sparsity is imposed on H to force ambiguous structure onto W

20 Informatics and Mathematical Modelling / Intelligent Signal Processing 20 Morten Mørup NMF2D SNMF2D Real music example of how imposing sparseness resolves the ambiguity between W and H

21 Informatics and Mathematical Modelling / Intelligent Signal Processing 21 Morten Mørup Unique!! Not unique PARAFAC (Harshman & Carrol and Chang 1970) Factor analysis (Charles Spearman ~1900) Extension to multi channel analysis by the PARAFAC model

22 Informatics and Mathematical Modelling / Intelligent Signal Processing 22 Morten Mørup cPARAFAC: Sparse Non-negative Tensor Factor 2D deconvolution (SNTF2D) (Extension of Fitzgerald et al. 2005, 2006 to form a sparse double deconvolution)

23 Informatics and Mathematical Modelling / Intelligent Signal Processing 23 Morten Mørup SNTF2D algorithms

24 Informatics and Mathematical Modelling / Intelligent Signal Processing 24 Morten Mørup Tchaikovsky: Violin Concert in D MajorMozart Sonate no. 16 in C Major

25 Informatics and Mathematical Modelling / Intelligent Signal Processing 25 Morten Mørup Stereo recording of ”Fog is Lifting” by Carl Nielsen

26 Informatics and Mathematical Modelling / Intelligent Signal Processing 26 Morten Mørup Applications –Source separation. –Music information retrieval. –Automatic music transcription (MIDI compression). –Source localization (beam forming)

27 Informatics and Mathematical Modelling / Intelligent Signal Processing 27 Morten Mørup PART II: Higher Order NMF (HONMF)

28 Informatics and Mathematical Modelling / Intelligent Signal Processing 28 Morten Mørup Higher Order Non-negative Matrix Factorization (HONMF) Motivation: Many of the data sets previously explored by the Tucker model are non-negative and could with good reason be decomposed under constraints of non-negativity on all modalities including the core. Spectroscopy data (Smilde et al. 1999,2004, Andersson & Bro 1998, Nørgard & Ridder 1994) Web mining (Sun et al., 2004) Image Analysis (Vasilescu and Terzopoulos, 2002, Wang and Ahuja, 2003, Jian and Gong, 2005) Semantic Differential Data (Murakami and Kroonenberg, 2003) And many more……

29 Informatics and Mathematical Modelling / Intelligent Signal Processing 29 Morten Mørup However, non-negative Tucker decompositions are not in general unique! But - Imposing sparseness overcomes this problem!

30 Informatics and Mathematical Modelling / Intelligent Signal Processing 30 Morten Mørup The Tucker Model

31 Informatics and Mathematical Modelling / Intelligent Signal Processing 31 Morten Mørup Algorithms for HONMF

32 Informatics and Mathematical Modelling / Intelligent Signal Processing 32 Morten Mørup Results HONMF with sparseness, above imposed on the core can be used for model selection -here indicating the PARAFAC model is the appropriate model to the data. Furthermore, the HONMF gives a more part based hence easy interpretable solution than the HOSVD.

33 Informatics and Mathematical Modelling / Intelligent Signal Processing 33 Morten Mørup Evaluation of uniqueness

34 Informatics and Mathematical Modelling / Intelligent Signal Processing 34 Morten Mørup Data of a Flow Injection Analysis (Nørrgaard, 1994) HONMF with sparse core and mixing captures unsupervised the true mixing and model order!

35 Informatics and Mathematical Modelling / Intelligent Signal Processing 35 Morten Mørup Conclusion HONMF not in general unique, however when imposing sparseness uniqueness can be achieved. Algorithms devised for LS and KL able to impose sparseness on any combination of modalities The HONMF decompositions more part based hence easier to interpret than other Tucker decompositions such as the HOSVD. Imposing sparseness can work as model selection turning of excess components

36 Informatics and Mathematical Modelling / Intelligent Signal Processing 36 Morten Mørup Coming soon in a MATLAB implementation near You

37 Informatics and Mathematical Modelling / Intelligent Signal Processing 37 Morten Mørup References Carroll, J. D. and Chang, J. J. Analysis of individual differences in multidimensional scaling via an N-way generalization of "Eckart-Young" decomposition, Psychometrika 35 1970 283--319 Eggert, J. and Korner, E. Sparse coding and NMF. In Neural Networks volume 4, pages 2529-2533, 2004 Eggert, J et al Transformation-invariant representation and nmf. In Neural Networks, volume 4, pages 535-2539, 2004 Fiitzgerald, D. et al. Non-negative tensor factorization for sound source separation. In proceedings of Irish Signals and Systems Conference, 2005 FitzGerald, D. and Coyle, E. C Sound source separation using shifted non.-negative tensor factorization. In ICASSP2006, 2006 Fitzgerald, D et al. Shifted non-negative matrix factorization for sound source separation. In Proceedings of the IEEE conference on Statistics in Signal Processing. 2005 Harshman, R. A. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modal factor analysis},UCLA Working Papers in Phonetics 16 1970 1—84 Harshman, Richard A.Harshman and Hong, Sungjin Lundy, Margaret E. Shifted factor analysis—Part I: Models and properties J. Chemometrics (17) pages 379–388, 2003 Kiers, Henk A. L. and Berge, Jos M. F. ten and Bro, Rasmus PARAFAC2 - Part I. A direct fitting algorithm for the PARAFAC2 model, Journal of Chemometrics (13) nr.3-4 pages 275-294, 1999 Lathauwer, Lieven De and Moor, Bart De and Vandewalle, Joos MULTILINEAR SINGULAR VALUE DECOMPOSITION.SIAM J. MATRIX ANAL. APPL.2000 (21)1253–1278 Lee, D.D. and Seung, H.S. Algorithms for non-negative matrix factorization. In NIPS, pages 556-462, 2000 Lee, D.D and Seung, H.S. Learning the parts of objects by non-negative matrix factorization, NATURE 1999 Murakami, Takashi and Kroonenberg, Pieter M. Three-Mode Models and Individual Differences in Semantic Differential Data, Multivariate Behavioral Research(38) no. 2 pages 247-283, 2003 Mørup, M. and Hansen, L.K.and Arnfred, S.M.Decomposing the time-frequency representation of EEG using nonnegative matrix and multi-way factorization Technical report, Institute for Mathematical Modeling, Technical University of Denmark, 2006a Mørup, M. and Schmidt, M.N. Sparse non-negative matrix factor 2-D deconvolution. Technical report, Institute for Mathematical Modeling, Tehcnical University of Denmark, 2006b Mørup, M and Schmidt, M.N. Non-negative Tensor Factor 2D Deconvolution for multi-channel time-frequency analysis. Technical report, Institute for Mathematical Modeling, Technical University of Denmark, 2006c Schmidt, M.N. and Mørup, M. Non-negative matrix factor 2D deconvolution for blind single channel source separation. In ICA2006, pages 700-707, 2006d Mørup, M. and Hansen, L.K.and Arnfred, S.M. Algorithms for Sparse Higher Order Non-negative Matrix Factorization (HONMF), Technical report, Institute for Mathematical Modeling, Technical University of Denmark, 2006e Nørgaard, L and Ridder, C.Rank annihilation factor analysis applied to flow injection analysis with photodiode-array detection Chemometrics and Intelligent Laboratory Systems 1994 (23) 107-114 Schmidt, M.N. and Mørup, M. Sparse Non-negative Matrix Factor 2-D Deconvolution for Automatic Transcription of Polyphonic Music, Technical report, Institute for Mathematical Modelling, Tehcnical University of Denmark, 2005 Smaragdis, P. Non-negative Matrix Factor deconvolution; Extraction of multiple sound sources from monophonic inputs. International Symposium on independent Component Analysis and Blind Source Separation (ICA)W Smilde, Age K. Smilde and Tauller, Roma and Saurina, Javier and Bro, Rasmus, Calibration methods for complex second-order data Analytica Chimica Acta 1999 237-251 Sun, Jian-Tao and Zeng, Hua-Jun and Liu, Huanand Lu Yuchang and Chen Zheng CubeSVD: a novel approach to personalized Web search WWW '05: Proceedings of the 14th international conference on World Wide Web pages 382—390, 2005 Tamara G. Kolda Multilinear operators for higher-order decompositions technical report Sandia national laboratory 2006 SAND2006-2081. Tucker, L. R. Some mathematical notes on three-mode factor analysis Psychometrika 31 1966 279—311 Welling, M. and Weber, M. Positive tensor factorization. Pattern Recogn. Lett. 2001 Vasilescu, M. A. O. and Terzopoulos, Demetri Multilinear Analysis of Image Ensembles: TensorFaces, ECCV '02: Proceedings of the 7th European Conference on Computer Vision-Part I, 2002


Download ppt "Informatics and Mathematical Modelling / Intelligent Signal Processing 1 Morten Mørup Extensions of Non-negative Matrix Factorization to Higher Order data."

Similar presentations


Ads by Google