SAND C 1/17 Coupled Matrix Factorizations using Optimization Daniel M. Dunlavy, Tamara G. Kolda, Evrim Acar Sandia National Laboratories SIAM Conference on Computational Science and Engineering March 4, 2009 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

2/17 Motivating Problems Data with multiple types of two-way relationships –Bibliometric analysis author-document, term-document, author-venue, etc. Can we predict potential co-authors? –Movie ratings movie-actor, user-movie, actor-award Can we predict useful movie ratings for other users? Consistent dimensionality reduction Improved interpretation through non-negativity constraints

3/17 Some Related Work Simultaneous factor analysis –Gramian matrices [Levin, 1966] –Test score covariance matrices over time [Millsap, et al., 1988] Simultaneous diagonalization –Population differentiation in biology [Thorpe, 1988] –Blind source separation [Ziehe et al., 2004] Generalized SVD Damped or constrained least squares [Van Loan, 1976] –Microarray data analysis [Alter, et al., 2003] –Multimicrophone speech filtering [Doclo and Moonen, 2002] Simultaneous Non-negative Matrix Factorization –Gene clustering in microarray data [Badea, 2007; 2008] Tensor decompositions –Data mining, chemometrics, neuroscience [Kolda, Acar, Bro, Park, Zhang, Berry, Chen, Martin, CSE09] matrices of same size only 2 matrices slow at least one common dimension

4/17 Coupled Non-negative Matrix Factorization (CNMF) Given Solve document-term document-author

5/17 Method: CNMF-ALS CNMF-ALS: Alternating Least Squares [Extends Berry, et al., 2006] linear least squares + simple projection to constraint boundary

6/17 Method: CNMF-MULT CNMF-MULT: Multiplicative Updates [Badea, 2007; Badea, 2008; extends Lee and Seung, 2001]

7/17 Method: CNMF-OPT CNMF-OPT: Projective Nonlinear CG, More-Thuente LS [Extends Acar, Kolda, and Dunlavy, 2009 and Lin, 2007]

8/17 Matlab Experiments Noise: mnpr*# var

9/17 Results: No noise, r = r*

10/17 Results: No noise, r = r*

11/17 Results: No noise, r = r*

12/17 Results: No noise, r = r*+1

13/17 Results: No noise, r=r*+1

14/17 Results: Noisy data, r=r*+1

15/17 Future Work Extending other promising methods to CNMF –Block principal pivoting based NMF [Park, et al. 2008] –Projected gradient NMF [Lin, 2007] –Projected Newton NMF [Kim, et al., 2008] CNMF-OPT extensions –Sparse data, regularization [Acar, Kolda, and Dunlavy, 2009] –Sparsity constraints [Park, et al. 2008] Numerical experiments –Scale to larger data sets –Comparisons on real data sets [Park, et al. 2008] Alternate models / problem formulations –Coupling matrix and tensor decompositions (CNMF/CNTF)

16/17 Conclusions Coupled matrix factorizations –Method for computing factorizations consistent along common dimensions in data Results – CNMF-OPT Fast and accurate –Overfactors well and handles noise well – CNMF-ALS Fast, but not accurate –Overfactoring is a big challenge – CNMF-MULT Accurate, but may be too slow (similar to NMF results) Future Work –Identified several promising paths forward

17/17 Thank You Coupled Matrix Factorizations using Optimization Danny Dunlavy