1. Evaluating Novelty and Diversity Charles Clarke School of Computer Science University of Waterloo two talks in one!

2. Goals for Evaluation Measures meaningful tractable reusable

3. Evaluation Framework We examine a framework for evaluation. Specific measures covered by the framework include: Clarke et al. (SIGIR 08) Agrawal et al. (WSDM 09) Clarke et al. (ICTIR 09)

4. Talk #1: Evaluating Diversity Charles Clarke School of Computer Science University of Waterloo

5. Query: windows 1.Microsoft Windows a)When will Windows 7 be released? b)Whats the Windows update URL? c)I want to download Windows Live Essentials 2.House windows a)Where can I buy replacement windows? b)What brands are available? c)Aluminum or vinyl? 3.Windows Restaurant, Las Vegas

6. Nuggets Nugget = any binary property of a document Provides address of a Pella dealer. Discusses history of the Windows OS. Is the Windows update page. (factual, topical and navigational) Problem: potentially thousands per query.

7. Evaluation Model user information needs using nuggets. Different users will be interested in different combinations of nuggets. Express judgments in terms of nuggets. Judgments may be automatic or manual. Judgments are binary: Does this document contain this nugget? Nuggets link users and documents

8. Interdependencies Problem: Complex interdependencies between nuggets. Three possible simplifying assumptions: 1.User interested in nugget A will always be interested in nugget B. 2.User interested in nugget A will never be interested in nugget B. 3.Nuggets A and B are independent.

9. Possible Assumption #1 If a user interested in nugget A will always be interested in nugget B, then A and B can be treated as the same nugget.

10. Possible Assumption #2 A user interested in nugget A will never be interested in nugget B (and vice versa). A users interest in nugget A depends on their interest in nugget B. Nugget A and nugget be may be viewed as representing different interpretations of the query.

11. Query: windows 1.Microsoft Windows a)When will Windows 7 be released? b)Whats the Windows update URL? c)I want to download Windows Live Essentials 2.House windows a)Where can I buy replacement windows? b)What brands are available? c)Aluminum or vinyl? 3.Windows Restaurant, Las Vegas

12. Query Interpretations Assume M interpretations Compute any effectiveness measure with respect to each interpretation (S j ) Compute weighted average (where p j is probability of interpretation j) Agrawal et al, 2009

13. Possible Assumption #3 A users interest in nugget A is independent of their interest in nugget B. The probability that the user is interested in nugget A is a constant (p A ). The probability that the user is interested in nugget B is a constant (p B ).

14. Query: windows 1.Microsoft Windows a)When will Windows 7 be released? b)Whats the Windows update URL? c)I want to download Windows Live Essentials 2.House windows a)Where can I buy replacement windows? b)What brands are available? c)Aluminum or vinyl? 3.Windows Restaurant, Las Vegas

15. Relevance framework A document is relevant if it contains any relevant information (with N nuggets).

16. Relevance Assume constant user probabilities Assume constant document probabilities J(d, i) = 1 iff document d is judged to contain nugget i count the nuggets

17. Probability of Relevance Estimated probability of relevance replaces relevance in standard evaluation measures, including nDCG, MAP, and Rank-biased precision. Assumptions #2 and #3 can then be combined. Other estimation methods possible.

18. Research Issues (talk #1) Identifying nuggets automatically –Clustering –Co-clicks –Query refinement Automatic judging –Patterns –Classification How many nuggets are enough? Estimating probability of relevance

19. Conclusions (talk #1) Evaluating diversity requires us to model and represent the diversity. Nuggets represent one possible solution. Simple user model; simple assumptions; simple judging.

20. Questions? Talk #1: Evaluating Diversity Charles Clarke School of Computer Science University of Waterloo

21. Intermission The TREC 2009 Web Track traditional adhoc task novelty and diversity task ClueWeb09 dataset (one billion pages) explore effectiveness measures http://plg.uwaterloo.ca/~trecweb

22. Intermission: Free sample topic physical therapist The user requires information regarding the profession and the services it provides. What does a physical therapist do? Where can I find a physical therapist? How much does physical therapy cost per hour? …

23. Talk #2: Evaluating Novelty Charles Clarke School of Computer Science University of Waterloo

24. Novelty Novelty depends on diversity. Previous talk considered probability of relevance in isolation (e.g., for the top- ranked document). In this talk we will examine how user context impacts the probability of relevance.

25. User context

26. Simplest context model Ranked list User scans result 1, 2, 3, 4, 5, … in order. Novelty of result k considered in light of the first k-1 results.

27. Relevance framework

28. Relevance Assuming constant probabilities.

29. Beyond the ranked list

30. Research issues (talk #2) Better user models Prior browsing context, local context, etc. Evaluating impact of result presentation methods –Better captions –Query suggestions –Instant answers (stock quotes, weather, product prices, definitions)

31. Conclusions (talk #2) Modeling and representing diversity allows us to consider novelty. User models should be simple enough to be tractable. User models should be complex enough to be meaningful.

32. Questions? Talk #2: Evaluating Novelty Charles Clarke School of Computer Science University of Waterloo