1. CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #21: Tensor decompositions C. Faloutsos

2. CMU SCS 2 Must-read Material Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. Technical Report SAND2007-6702, Sandia National Laboratories, Albuquerque, NM and Livermore, CA, November 2007 Tensor decompositions and applications.

3. CMU SCS 3 Outline Goal: ‘Find similar / interesting things’ Intro to DB Indexing - similarity search Data Mining

4. CMU SCS 4 Indexing - Detailed outline primary key indexing secondary key / multi-key indexing spatial access methods fractals text Singular Value Decomposition (SVD) -… -Tensors multimedia...

5. CMU SCS 5 Most of foils by Dr. Tamara Kolda (Sandia N.L.) csmr.ca.sandia.gov/~tgkolda Dr. Jimeng Sun (CMU -> IBM) www.cs.cmu.edu/~jimeng 3h tutorial: www.cs.cmu.edu/~christos/TALKS/SDM-tut-07/

6. CMU SCS 6 Outline Motivation - Definitions Tensor tools Case studies

7. CMU SCS 7 Motivation 0: Why “matrix”? Why matrices are important?

8. CMU SCS 8 Examples of Matrices: Graph - social network John PeterMaryNick... John Peter Mary Nick... 0112255... 5067

9. CMU SCS 9 Examples of Matrices: cloud of n-d points chol#blood# age..... John Peter Mary Nick...

10. CMU SCS 10 Examples of Matrices: Market basket market basket as in Association Rules milkbreadchoc.wine... John Peter Mary Nick...

11. CMU SCS 11 Examples of Matrices: Documents and terms Paper#1 Paper#2 Paper#3 Paper#4 dataminingclassif.tree...

12. CMU SCS 12 Examples of Matrices: Authors and terms dataminingclassif.tree... John Peter Mary Nick...

13. CMU SCS 13 Examples of Matrices: sensor-ids and time-ticks t1 t2 t3 t4 temp1temp2humid.pressure...

14. CMU SCS 14 Motivation: Why tensors? Q: what is a tensor?

15. CMU SCS 15 Motivation 2: Why tensor? A: N-D generalization of matrix: dataminingclassif.tree... John Peter Mary Nick... KDD’07

16. CMU SCS 16 Motivation 2: Why tensor? A: N-D generalization of matrix: dataminingclassif.tree... John Peter Mary Nick... KDD’06 KDD’05 KDD’07

17. CMU SCS 17 Tensors are useful for 3 or more modes Terminology: ‘mode’ (or ‘aspect’): dataminingclassif.tree... Mode (== aspect) #1 Mode#2 Mode#3

18. CMU SCS 18 Motivating Applications Why matrices are important? Why tensors are useful? –P1: social networks –P2: web mining

19. CMU SCS 19 P1: Social network analysis Traditionally, people focus on static networks and find community structures We plan to monitor the change of the community structure over time

20. CMU SCS 20 P2: Web graph mining How to order the importance of web pages? –Kleinberg’s algorithm HITS –PageRank –Tensor extension on HITS (TOPHITS) context-sensitive hypergraph analysis

21. CMU SCS 21 Outline Motivation – Definitions Tensor tools Case studies Tensor Basics Tucker PARAFAC

22. CMU SCS Tensor Basics

23. CMU SCS 23 Reminder: SVD –Best rank-k approximation in L2 A m n  m n U VTVT 

24. CMU SCS 24 Reminder: SVD –Best rank-k approximation in L2 A m n  + 1u1v11u1v1 2u2v22u2v2

25. CMU SCS 25 Goal: extension to >=3 modes ~ I x R K x R A B J x R C R x R x R I x J x K +…+=

26. CMU SCS 26 Main points: 2 major types of tensor decompositions: PARAFAC and Tucker both can be solved with ``alternating least squares’’ (ALS) Details follow

27. CMU SCS Specially Structured Tensors

28. CMU SCS 28 = U I x R V J x R W K x R R x R x R Specially Structured Tensors Tucker TensorKruskal Tensor I x J x K = U I x R V J x S W K x T R x S x T I x J x K Our Notation +…+= u1u1 uRuR v1v1 w1w1 vRvR wRwR “core”

29. CMU SCS 29 Specially Structured Tensors Tucker TensorKruskal Tensor In matrix form: details

30. CMU SCS Tensor Decompositions

31. CMU SCS 31 Tucker Decomposition - intuition I x J x K ~ A I x R B J x S C K x T R x S x T author x keyword x conference A: author x author-group B: keyword x keyword-group C: conf. x conf-group : how groups relate to each other Needs elaboration!

32. CMU SCS 32 Intuition behind core tensor 2-d case: co-clustering [Dhillon et al. Information-Theoretic Co- clustering, KDD’03]

33. CMU SCS 33 m m n nl k k l eg, terms x documents

34. CMU SCS 34 term x term-group doc x doc group term group x doc. group med. terms cs terms common terms med. doc cs doc

35. CMU SCS 35

36. CMU SCS 36 Tucker Decomposition Proposed by Tucker (1966) AKA: Three-mode factor analysis, three-mode PCA, orthogonal array decomposition A, B, and C generally assumed to be orthonormal (generally assume they have full column rank) is not diagonal Not unique Recall the equations for converting a tensor to a matrix I x J x K ~ A I x R B J x S C K x T R x S x T Given A, B, C, the optimal core is:

37. CMU SCS 37 Outline Motivation – Definitions Tensor tools Case studies Tensor Basics Tucker PARAFAC

38. CMU SCS 38 CANDECOMP/PARAFAC Decomposition CANDECOMP = Canonical Decomposition (Carroll & Chang, 1970) PARAFAC = Parallel Factors (Harshman, 1970) Core is diagonal (specified by the vector ) Columns of A, B, and C are not orthonormal If R is minimal, then R is called the rank of the tensor (Kruskal 1977) Can have rank( ) > min{I,J,K} ¼ I x R K x R A B J x R C R x R x R I x J x K +…+=

39. CMU SCS 39 Tucker vs. PARAFAC Decompositions Tucker –Variable transformation in each mode –Core G may be dense –A, B, C generally orthonormal –Not unique PARAFAC –Sum of rank-1 components –No core, i.e., superdiagonal core –A, B, C may have linearly dependent columns –Generally unique I x J x K ~ A I x R B J x S C K x T R x S x T I x J x K +…+ ~ a1a1 aRaR b1b1 c1c1 bRbR cRcR IMPORTANT

40. CMU SCS 40 Tensor tools - summary Two main tools –PARAFAC –Tucker Both find row-, column-, tube-groups –but in PARAFAC the three groups are identical To solve: Alternating Least Squares Toolbox: from Tamara Kolda: http://csmr.ca.sandia.gov/~tgkolda/TensorToolbox/

41. CMU SCS 41 Outline Motivation - Definitions Tensor tools Case studies

42. CMU SCS 42 P1: Web graph mining How to order the importance of web pages? –Kleinberg’s algorithm HITS –PageRank –Tensor extension on HITS (TOPHITS)

43. CMU SCS 43 Kleinberg’s Hubs and Authorities (the HITS method) Sparse adjacency matrix and its SVD: authority scores for 1 st topic hub scores for 1 st topic hub scores for 2 nd topic authority scores for 2 nd topic from to Kleinberg, JACM, 1999

44. CMU SCS 44 authority scores for 1 st topic hub scores for 1 st topic hub scores for 2 nd topic authority scores for 2 nd topic from to HITS Authorities on Sample Data We started our crawl from http://www-neos.mcs.anl.gov/neos, and crawled 4700 pages, resulting in 560 cross-linked hosts.

45. CMU SCS 45 Three-Dimensional View of the Web Observe that this tensor is very sparse! Kolda, Bader, Kenny, ICDM05

46. CMU SCS 46 Topical HITS (TOPHITS) Main Idea: Extend the idea behind the HITS model to incorporate term (i.e., topical) information. authority scores for 1 st topic hub scores for 1 st topic hub scores for 2 nd topic authority scores for 2 nd topic from to term term scores for 1 st topic term scores for 2 nd topic

47. CMU SCS 47 Topical HITS (TOPHITS) Main Idea: Extend the idea behind the HITS model to incorporate term (i.e., topical) information. authority scores for 1 st topic hub scores for 1 st topic hub scores for 2 nd topic authority scores for 2 nd topic from to term term scores for 1 st topic term scores for 2 nd topic

48. CMU SCS 48 TOPHITS Terms & Authorities on Sample Data TOPHITS uses 3D analysis to find the dominant groupings of web pages and terms. authority scores for 1 st topic hub scores for 1 st topic hub scores for 2 nd topic authority scores for 2 nd topic from to term term scores for 1 st topic term scores for 2 nd topic Tensor PARAFAC w k = # unique links using term k

49. CMU SCS GigaTensor: Scaling Tensor Analysis Up By 100 Times – Algorithms and Discoveries U Kang Christos Faloutsos School of Computer Science Carnegie Mellon University Evangelos Papalexakis Abhay Harpale

50. CMU SCS P2: N.E.L.L. analysis NELL: Never Ending Language Learner –Q1: dominant concepts / topics? –Q2: synonyms for a given new phrase? 50 “Barrack Obama is the president of U.S.” “Eric Clapton plays guitar” (26M) (48M) NELL (Never Ending Language Learner) Nonzeros =144M (26M)

51. CMU SCS A1: Concept Discovery Concept Discovery in Knowledge Base

52. CMU SCS A1: Concept Discovery

53. CMU SCS A2: Synonym Discovery Synonym Discovery in Knowledge Base a1a1 a2a2 aRaR … (Given) subject (Discovered) synonym 1 (Discovered) synonym 2

54. CMU SCS A2: Synonym Discovery

55. CMU SCS 55 Conclusions Real data are often in high dimensions with multiple aspects (modes) Matrices and tensors provide elegant theory and algorithms I x J x K +…+ ~ a1a1 aRaR b1b1 c1c1 bRbR cRcR

56. CMU SCS 56 References Inderjit S. Dhillon, Subramanyam Mallela, Dharmendra S. Modha: Information-theoretic co-clustering. KDD 2003: 89-98 T. G. Kolda, B. W. Bader and J. P. Kenny. Higher- Order Web Link Analysis Using Multilinear Algebra. In: ICDM 2005, Pages 242-249, November 2005. Jimeng Sun, Spiros Papadimitriou, Philip Yu. Window- based Tensor Analysis on High-dimensional and Multi- aspect Streams, Proc. of the Int. Conf. on Data Mining (ICDM), Hong Kong, China, Dec 2006