Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus

Similar presentations


Presentation on theme: "Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus"— Presentation transcript:

1 Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus

2 Recommender systems What are they? Example: Amazon

3 Controversial recommenders What to do when your TiVo thinks youre gay, Wall Street Journal, Nov. 26, 2002

4 Controversial recommenders What to do when your TiVo thinks youre gay, Wall Street Journal, Nov. 26, 2002

5 Controversial recommenders What to do when your TiVo thinks youre gay, Wall Street Journal, Nov. 26, 2002

6 Controversial recommenders Wal-Mart DVD recommendations

7 Controversial recommenders Wal-Mart DVD recommendations

8 Controversial recommenders Wal-Mart DVD recommendations

9 Googles mission To organize the world's information and make it universally accessible and useful.

10 communities

11 Community recommender Goal: Per-community ranked recommendations How to determine?

12 Community recommender Goal: Per-community ranked recommendations How to determine? –Implicit collaborative filtering –Look for common membership between pairs of communities

13 Terminology Consider each community to be a set of members –B: base community (e.g., Pizza) –R: related community (e.g., Cheese) Similarity measure –Based on overlap |BR|

14 Example: Pizza

15

16 Terminology Consider each community to be a set of members –B: base community (e.g., Wine) –R: related community (e.g., Linux) Similarity measure –Based on overlap |BR| –Also depends on |B| and |R| –Possibly asymmetric

17 Example of asymmetry

18 Similarity measures L1 normalization L2 normalization Pointwise mutual information –Positive correlations –Positive and negative correlations Salton tf-idf Log-odds

19 L1 normalization Vector notation Set notation

20 L2 normalization Vector notation Set notation

21 Mutual information: positive correlation Formally, Informally, how well membership in the base community predicts membership in the related community

22 Mutual information: positive and negative correlation

23 Salton tf-idf

24 LogOdds0 Formally, Informally, how much likelier a member of B is to belong to R than a non-member of B is.

25 LogOdds0 Formally, Informally, how much likelier a member of B is to belong to R than a non-member of B is. This yielded the same rankings as L1.

26 LogOdds

27 Predictions? Were there significant differences among the measures? –Top-ranked recommendations –User preference Which measure was best? Was there a partial or total ordering of measures?

28 Recommendations for I love wine (2400)

29 Experiment Precomputed top 12 recommendations for each base community for each similarity measure When a user views a community page –Hash the community and user ID to –Select an ordered pair of measures to –Interleave, filtering out duplicates Track clicks of new users

30 Click interpretation

31

32 Overall click rate (July 1-18) Total recommendation pages generated: 4,106,050

33 Overall click rate (July 1-18)

34

35 Analysis For each pair of similarity measures M a and M b and each click C, either: M a recommended C more highly than M b M a and M b recommended C equally M b recommended C more highly than M a

36 Results Clicks leading to joins L2 » MI1 » MI2 » IDF L1 » LogOdds All clicks L2 » L1 » MI1 » MI2 IDF» LogOdds

37 Positional effects Original experiment –Ordered recommendations by rank Second experiment –Generated recommendations using L2 –Pseudo-randomly ordered recommendations, tracking clicks by placement –Tracked 1.3 M clicks between September 22-October 21

38 Results: single row (n=28108) Namorado Para o Bulldog

39 Results: single row (n=28,108) p=.12 (not significant)

40 Results: two rows (n=24,459)

41 p <.001

42 Results: 3 rows (n=1,226,659)

43 p <.001

44 Users reactions Hundreds of requests per day to add recommendations Angry requests from community creators –General –Specific

45 Amusing recommendations C++

46 Amusing recommendations C++Whats she trying to say? For every time a woman has confused you…

47 Amusing recommendations Chocolate

48 Amusing recommendations ChocolatePMS

49 Allowing community owners to set recommendations

50

51 Manual recommendations Eight days after release –50,876 community owners –Added 267,623 recommendations –Deleted 59,599 recommendations –Affecting 73,230 base communities and –111,936 related communities Open question: How do they compare with automatic recommendations?

52 Future research 1 Determining similar users based on common communities –Is it useful? –Will the measures make the same total order? (9 users)

53 Other types of information Distance in social network Demographic –Country –Age –Etc.

54 Future research 2 Per-user community recommendations –Using social network information –Using profile information (e.g., country)

55 Future research 2 Per-user community recommendations –Using social network information –Using profile information (e.g., country)

56 Future research 2 Per-user community recommendations –Using social network information –Using profile information (e.g., country)

57 Future research 3 Do we get the same ordering for other domains? L2 » MI1 » MI2 » IDF L1 » LogOdds

58 Acknowledgments Mehran Sahami Orkut Buyukkokten orkut team

59 Bonus material

60 Self-rated beauty beauty contest winners very attractive attractive average mirror-cracking material

61 Self-rated beauty: men beauty contest winners8% very attractive18% attractive39% average24% mirror-cracking material11%

62 Self-rated beauty: women beauty contest winners8% very attractive16% attractive39% average27% mirror-cracking material9%

63 Self-rated beauty by country Most beautiful –men: –women: Least beautiful –men: –women:

64 Self-rated beauty by country Most beautiful –men: Syrian –women: Barbadian Least beautiful –men: Gambian –women: Ascension Islanders

65 Ratings by others Karma –trustiness –sexiness –coolness How do these correlate with age?

66 Ratings by others

67 Friend counts

68 Self-rated best body part


Download ppt "Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus"

Similar presentations


Ads by Google