1. Personalizing Web Search Jaime Teevan, MIT with Susan T. Dumais and Eric Horvitz, MSR

3. Demo

4. Personalizing Web Search Motivation Algorithms Results Future Work

5. Personalizing Web Search Motivation Algorithms Results Future Work

6. Study of Personal Relevancy 15 SIS users x ~10 queries Evaluate 50 results  Highly relevant / Relevant / Irrelevant Query selection  Previously issued query  Chose from 10 pre-selected queries Collected evaluations for 137 queries  53 of pre-selected queries (2-9/query)

7. Relevant Results Have Low Rank Highly Relevant Relevant Irrelevant

8. Same Query, Different Intent Different meanings  “Information about the astronomical/astrological sign of cancer”  “information about cancer treatments” Different intents  “is there any new tests for cancer?”  “information about cancer treatments”

9. Same Intent, Different Evaluation Query: Microsoft  “information about microsoft, the company”  “Things related to the Microsoft corporation”  “Information on Microsoft Corp” 31/50 rated as not irrelevant  Only 6/31 do more than one agree  All three agree only for www.microsoft.com

10. More to Understand Do people cluster?  Even if they can’t state their intention How are the differences reflected?  Can they be seen from the information on a person’s computer? Can we do better than the ranking that would make everyone the most happy?  Best common ranking: +38%  Best personalized ranking: +55%

11. Personalizing Web Search Motivation Algorithms Results Future Work

12. Personalization Algorithms Standard IR Related to relevance feedback Query expansion Document Query User Server Client v. Result re-ranking

13. Result Re-Ranking Takes full advantage of SIS Ensures privacy Good evaluation framework Look at light weight user models  Collected on server side  Sent as query expansion

14. BM25 N nini NniNni w i = log riri R with Relevance Feedback Score = Σ tf i * w i

15. BM25 with Relevance Feedback N nini (r i +0.5)(N-n i -R+r i +0.5) (n i -r i +0.5)(R-r i +0.5) riri R w i = log Score = Σ tf i * w i

16. (r i +0.5)(N-n i -R+r i +0.5) (n i - r i +0.5)(R-r i +0.5) User Model as Relevance Feedback N nini R riri Score = Σ tf i * w i (r i +0.5)(N’-n i ’-R+r i +0.5) (n i ’- r i +0.5)(R-r i +0.5) w i = log N’ = N+R n i ’ = n i +ri

17. User Model as Relevance Feedback N nini R riri World User Score = Σ tf i * w i

18. User Model as Relevance Feedback R riri User N nini World World related to query N nini Score = Σ tf i * w i

19. User Model as Relevance Feedback N nini R riri World User World related to query User related to query R N nini riri Query Focused Matching Score = Σ tf i * w i

20. User Model as Relevance Feedback N nini R riri World User Web related to query User related to query R N riri Query Focused Matching nini World Focused Matching Score = Σ tf i * w i

21. Parameters Matching User representation World representation Query expansion

22. Parameters Matching User representation World representation Query expansion Query focused World focused

23. Parameters Matching User representation World representation Query expansion Query focused World focused

24. User Representation Stuff I’ve Seen (SIS) index Recently indexed documents Web documents in SIS index Query history Relevance judgments None

25. Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None

26. Parameters Matching User representation World representation Query expansion Query Focused World Focused All SIS Recent SIS Web SIS Query History Relevance Feedback None

27. World Representation Document Representation  Full text  Title and snippet Corpus Representation  Web  Result set – title and snippet  Result set – full text

28. Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet

29. Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet

30. Query Expansion All words in document Query focused The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through...

31. Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

32. Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

33. Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

34. Personalizing Web Search Motivation Algorithms Results Future Work

35. Baselines Best possible Random Text based ranking Web ranking URL Boost http://mail.yahoo.com/inbox/msg10 +1

36. Best Parameter Settings Richer user representation better  SIS > Recent > Web > Query History > None Suggests rich client important Efficiency hacks don’t hurt  Snippets query focused  Length normalization not an issue Query focus good

37. Text Alone Not Enough Better than some baselines  Better than random  Better than no user representation  Better than relevance feedback Worse than Web results Blend in other features  Web ranking  URL boost

38. Good, but Lots of Room to Grow Best combination: 9.1% improvement Best possible: 51.5% improvement Assumes best Web combination selected Only improves results 2/3 of the time

39. Personalizing Web Search Motivation Algorithms Results Future Work

40. Finding the Best Parameter Setting Almost always some parameter setting that improves results Use learning to select parameters  Based on individual  Based on query  Based on results Give user control?

41. Further Exploration of Algorithms Larger parameter space to explore  More complex user model subsets  Different parsing (e.g., phrases)  Tune BM25 parameters What is really helping?  Generic user model or personal model  Use different indices for the queries Deploy system

42. Practical Issues Efficiency issues  Can interfaces mitigate some of the issues? Merging server and client  Query expansion Get more relevant results in the set to be re-ranked  Design snippets for personalization

43. Thank you!