1. Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman Stanford University http://www.mmds.org Note to other teachers and users of these slides: We would be delighted if you found this our material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. If you make use of a significant portion of these slides in your own lecture, please include this message, or a link to our web site: http://www.mmds.orghttp://www.mmds.org

2. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org2

3. 134 355 455 3 3 222 5 211 3 3 1 480,000 users 17,700 movies 3J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org Matrix R

4. 134 355 455 3 3 2?? ? 21? 3 ? 1 Test Data Set 4 480,000 users 17,700 movies Predicted rating True rating of user x on item i Matrix R Training Data Set J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

5.  The winner of the Netflix Challenge!  Multi-scale modeling of the data: Combine top level, “regional” modeling of the data, with a refined, local view:  Global:  Overall deviations of users/movies  Factorization:  Addressing “regional” effects  Collaborative filtering:  Extract local patterns J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org5 Global effects Factorization Collaborative filtering

6.  Global:  Mean movie rating: 3.7 stars  The Sixth Sense is 0.5 stars above avg.  Joe rates 0.2 stars below avg.  Baseline estimation: Joe will rate The Sixth Sense 4 stars  Local neighborhood (CF/NN):  Joe didn’t like related movie Signs   Final estimate: Joe will rate The Sixth Sense 3.8 stars J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org6

7.  Earliest and most popular collaborative filtering method  Derive unknown ratings from those of “similar” movies (item-item variant)  Define similarity measure s ij of items i and j  Select k-nearest neighbors, compute the rating  N(i; x): items most similar to i that were rated by x 7J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org s ij … similarity of items i and j r xj …rating of user x on item j N(i;x)… set of items similar to item i that were rated by x

8.  In practice we get better estimates if we model deviations: J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org8 μ = overall mean rating b x = rating deviation of user x = (avg. rating of user x) – μ b i = (avg. rating of movie i) – μ Problems/Issues: 1) Similarity measures are “arbitrary” 2) Pairwise similarities neglect interdependencies among users 3) Taking a weighted average can be restricting Solution: Instead of s ij use w ij that we estimate directly from data ^ baseline estimate for r xi

9. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org9

10. 10 Why is this a good idea?

11.  Goal: Make good recommendations  Quantify goodness using RMSE: Lower RMSE  better recommendations  Want to make good recommendations on items that user has not yet seen. Can’t really do this!  Let’s set build a system such that it works well on known (user, item) ratings And hope the system will also predict well the unknown ratings 11J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 134 355 455 3 3 222 5 211 3 3 1

12. 12 Predicted rating True rating

13. 13J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

14. 14  … learning rate while |w new - w old | > ε: w old = w new w new = w old -  ·  w old

15. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org15 Global effects Factorization CF/NN

16. Grand Prize: 0.8563 Netflix: 0.9514 Movie average: 1.0533 User average: 1.0651 Global average: 1.1296 Basic Collaborative filtering: 0.94 CF+Biases+learned weights: 0.91 J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org16

17. Geared towards females Geared towards males Serious Funny 17J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Princess Diaries The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility

18.  “SVD” on Netflix data: R ≈ Q · P T  For now let’s assume we can approximate the rating matrix R as a product of “thin” Q · P T  R has missing entries but let’s ignore that for now!  Basically, we will want the reconstruction error to be small on known ratings and we don’t care about the values on the missing ones J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org18 45531 312445 53432142 24542 522434 42331.2-.4.1.5.6-.5.5.3-.2.32.11.1 -22.1-.7.3.7 -.92.41.4.3-.4.8-.5-2.5.3-.21.1 1.3-.11.2-.72.91.4.31.4.5.7-.8.1-.6.7.8.4-.3.92.41.7.6-.42.1 ≈ users items PTPT Q users R SVD: A = U  V T factors

19.  How to estimate the missing rating of user x for item i? J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org19 45531 312445 53432142 24542 522434 42331 items.2-.4.1.5.6-.5.5.3-.2.32.11.1 -22.1-.7.3.7 -.92.41.4.3-.4.8-.5-2.5.3-.21.1 1.3-.11.2-.72.91.4.31.4.5.7-.8.1-.6.7.8.4-.3.92.41.7.6-.42.1 ≈ items users ? PTPT q i = row i of Q p x = column x of P T factors Q

20.  How to estimate the missing rating of user x for item i? J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org20 45531 312445 53432142 24542 522434 42331 items.2-.4.1.5.6-.5.5.3-.2.32.11.1 -22.1-.7.3.7 -.92.41.4.3-.4.8-.5-2.5.3-.21.1 1.3-.11.2-.72.91.4.31.4.5.7-.8.1-.6.7.8.4-.3.92.41.7.6-.42.1 ≈ items users ? PTPT factors Q q i = row i of Q p x = column x of P T

21.  How to estimate the missing rating of user x for item i? J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org21 45531 312445 53432142 24542 522434 42331 items.2-.4.1.5.6-.5.5.3-.2.32.11.1 -22.1-.7.3.7 -.92.41.4.3-.4.8-.5-2.5.3-.21.1 1.3-.11.2-.72.91.4.31.4.5.7-.8.1-.6.7.8.4-.3.92.41.7.6-.42.1 ≈ items users ? Q PTPT 2.4 f factors q i = row i of Q p x = column x of P T

22. Geared towards females Geared towards males Serious Funny 22J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Princess Diaries The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility Factor 1 Factor 2

23. Geared towards females Geared towards males Serious Funny 23J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Princess Diaries The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility Factor 1 Factor 2

24.  Remember SVD:  A: Input data matrix  U: Left singular vecs  V: Right singular vecs   : Singular values  So in our case: “SVD” on Netflix data: R ≈ Q · P T A = R, Q = U, P T =  V T A m n  m n VTVT  24 U J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

25. 25J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

26. 26 45531 312445 53432142 24542 522434 42331.2-.4.1.5.6-.5.5.3-.2.32.11.1 -22.1-.7.3.7 -.92.41.4.3-.4.8-.5-2.5.3-.21.1 1.3-.11.2-.72.91.4.31.4.5.7-.8.1-.6.7.8.4-.3.92.41.7.6-.42.1  PTPT Q users items factors items users

28. 28 45531 312445 53432142 24542 522434 42331.2-.4.1.5.6-.5.5.3-.2.32.11.1 -22.1-.7.3.7 -.92.41.4.3-.4.8-.5-2.5.3-.21.1 1.3-.11.2-.72.91.4.31.4.5.7-.8.1-.6.7.8.4-.3.92.41.7.6-.42.1  PTPT Q users items factors items users J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

29.  Want to minimize SSE for unseen test data  Idea: Minimize SSE on training data  Want large k (# of factors) to capture all the signals  But, SSE on test data begins to rise for k > 2  This is a classical example of overfitting:  With too much freedom (too many free parameters) the model starts fitting noise  That is it fits too well the training data and thus not generalizing well to unseen test data 29 134 355 455 3 3 2?? ? 21? 3 ? 1 J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

30.  To solve overfitting we introduce regularization:  Allow rich model where there are sufficient data  Shrink aggressively where data are scarce 30 134 355 455 3 3 2?? ? 21? 3 ? 1 1, 2 … user set regularization parameters “error” “length” Note: We do not care about the “raw” value of the objective function, but we care in P,Q that achieve the minimum of the objective J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

31. Geared towards females Geared towards males serious funny 31J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Princess Diaries The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility Factor 1 Factor 2 min factors “error” + “length”

32. Geared towards females Geared towards males serious funny 32J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility Factor 1 Factor 2 The Princess Diaries min factors “error” + “length”

33. Geared towards females Geared towards males serious funny 33J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility Factor 1 Factor 2 min factors “error” + “length” The Princess Diaries

34. Geared towards females Geared towards males serious funny 34J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org The Lion King Braveheart Lethal Weapon Independence Day Amadeus The Color Purple Dumb and Dumber Ocean’s 11 Sense and Sensibility Factor 1 Factor 2 The Princess Diaries min factors “error” + “length”

35. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org35 How to compute gradient of a matrix? Compute gradient of every element independently!

36. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org36

37.  Convergence of GD vs. SGD J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org37 Iteration/step Value of the objective function GD improves the value of the objective function at every step. SGD improves the value but in a “noisy” way. GD takes fewer steps to converge but each step takes much longer to compute. In practice, SGD is much faster!

38. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org38

39. Koren, Bell, Volinksy, IEEE Computer, 2009 39J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

41. 41  μ = overall mean rating  b x = bias of user x  b i = bias of movie i user-movie interactionmovie biasuser bias User-Movie interaction  Characterizes the matching between users and movies  Attracts most research in the field  Benefits from algorithmic and mathematical innovations Baseline predictor  Separates users and movies  Benefits from insights into user’s behavior  Among the main practical contributions of the competition

42.  We have expectations on the rating by user x of movie i, even without estimating x’s attitude towards movies like i – Rating scale of user x – Values of other ratings user gave recently (day-specific mood, anchoring, multi-user accounts) – (Recent) popularity of movie i – Selection bias; related to number of ratings user gave on the same day (“frequency”) 42J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

43.  Example:  Mean rating:  = 3.7  You are a critical reviewer: your ratings are 1 star lower than the mean: b x = -1  Star Wars gets a mean rating of 0.5 higher than average movie: b i = + 0.5  Predicted rating for you on Star Wars: = 3.7 - 1 + 0.5 = 3.2 J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org43 Overall mean rating Bias for user x Bias for movie i User-Movie interaction

44.  Solve:  Stochastic gradient decent to find parameters  Note: Both biases b x, b i as well as interactions q i, p x are treated as parameters (we estimate them) J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org44 regularization goodness of fit is selected via grid- search on a validation set

45. 45 J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

46. Grand Prize: 0.8563 Netflix: 0.9514 Movie average: 1.0533 User average: 1.0651 Global average: 1.1296 Basic Collaborative filtering: 0.94 Latent factors: 0.90 Latent factors+Biases: 0.89 Collaborative filtering++: 0.91 J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org46

48.  Sudden rise in the average movie rating (early 2004)  Improvements in Netflix  GUI improvements  Meaning of rating changed  Movie age  Users prefer new movies without any reasons  Older movies are just inherently better than newer ones J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org48 Y. Koren, Collaborative filtering with temporal dynamics, KDD ’09

49.  Original model: r xi =  +b x + b i + q i · p x  Add time dependence to biases: r xi =  +b x (t)+ b i (t) +q i · p x  Make parameters b x and b i to depend on time  (1) Parameterize time-dependence by linear trends (2) Each bin corresponds to 10 consecutive weeks  Add temporal dependence to factors  p x (t)… user preference vector on day t J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org49 Y. Koren, Collaborative filtering with temporal dynamics, KDD ’09

50. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org50

51. Grand Prize: 0.8563 Netflix: 0.9514 Movie average: 1.0533 User average: 1.0651 Global average: 1.1296 Basic Collaborative filtering: 0.94 Latent factors: 0.90 Latent factors+Biases: 0.89 Collaborative filtering++: 0.91 J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org51 Latent factors+Biases+Time: 0.876 Still no prize!  Getting desperate. Try a “kitchen sink” approach!

52. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org52

53. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org53 June 26 th submission triggers 30-day “last call”

54.  Ensemble team formed  Group of other teams on leaderboard forms a new team  Relies on combining their models  Quickly also get a qualifying score over 10%  BellKor  Continue to get small improvements in their scores  Realize that they are in direct competition with Ensemble  Strategy  Both teams carefully monitoring the leaderboard  Only sure way to check for improvement is to submit a set of predictions  This alerts the other team of your latest score 54J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

55.  Submissions limited to 1 a day  Only 1 final submission could be made in the last 24h  24 hours before deadline…  BellKor team member in Austria notices (by chance) that Ensemble posts a score that is slightly better than BellKor’s  Frantic last 24 hours for both teams  Much computer time on final optimization  Carefully calibrated to end about an hour before deadline  Final submissions  BellKor submits a little early (on purpose), 40 mins before deadline  Ensemble submits their final entry 20 mins later  ….and everyone waits…. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org55

56. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org56

57. J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org57

58.  Some slides and plots borrowed from Yehuda Koren, Robert Bell and Padhraic Smyth  Further reading:  Y. Koren, Collaborative filtering with temporal dynamics, KDD ’09  http://www2.research.att.com/~volinsky/netflix/bpc.html http://www2.research.att.com/~volinsky/netflix/bpc.html  http://www.the-ensemble.com/ http://www.the-ensemble.com/ J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org58