1. Search and Retrieval: Relevance and Evaluation Prof. Marti Hearst SIMS 202, Lecture 20

2. Marti A. Hearst SIMS 202, Fall 1997 Finding Out About n Three phases: n Asking of a question n Construction of an answer n Assessment of the answer n Part of an iterative process

3. Marti A. Hearst SIMS 202, Fall 1997 Today n Relevance n Evaluation of IR Systems n Precision vs. Recall n Cutoff Points n Test Collections/TREC n Blair & Maron Study

4. Marti A. Hearst SIMS 202, Fall 1997 Evaluation n Why Evaluate? n What to Evaluate? n How to Evaluate?

5. Marti A. Hearst SIMS 202, Fall 1997 Why Evaluate?

6. Marti A. Hearst SIMS 202, Fall 1997 Why Evaluate? n Determine if the system is desirable n Make comparative assessments n Others?

7. Marti A. Hearst SIMS 202, Fall 1997 What to Evaluate?

8. Marti A. Hearst SIMS 202, Fall 1997 What to Evaluate? n How much learned about the collection n How much learned about a topic n How much of the information need is satisfied n How inviting the system is

9. Marti A. Hearst SIMS 202, Fall 1997 What to Evaluate? n What can be measured that reflects users’ ability to use system? (Cleverdon 66) n Coverage of Information n Form of Presentation n Effort required/Ease of Use n Time and Space Efficiency n Recall n proportion of relevant material actually retrieved n Precision n proportion of retrieved material actually relevant effectiveness

10. Marti A. Hearst SIMS 202, Fall 1997 Assessing the Answer n How well does it answer the question? n Complete answer? Partial? n Background Information? n Hints for further exploration? n How relevant is it to the user?

11. Marti A. Hearst SIMS 202, Fall 1997 Relevance n Subjective n Measurable to some extent n How often do people agree a document is relevant to a query

12. Marti A. Hearst SIMS 202, Fall 1997 Relevance n In what ways can a document be relevant to a query? n Answer precise question precisely. n Partially answer question. n Suggest a source for more information. n Give background information. n Remind the user of other knowledge. n Others...

13. Marti A. Hearst SIMS 202, Fall 1997 Retrieved vs. Relevant Documents Retrieved

14. Marti A. Hearst SIMS 202, Fall 1997 Retrieved vs. Relevant Documents Relevant Retrieved

15. Marti A. Hearst SIMS 202, Fall 1997 Retrieved vs. Relevant Documents Relevant High Precision Retrieved

16. Marti A. Hearst SIMS 202, Fall 1997 Retrieved vs. Relevant Documents Relevant High Recall Retrieved

17. Marti A. Hearst SIMS 202, Fall 1997 Retrieved vs. Relevant Documents Relevant High Precision High Recall Retrieved

18. Marti A. Hearst SIMS 202, Fall 1997 Retrieved vs. Relevant Documents Relevant High Precision High Recall Retrieved

19. Marti A. Hearst SIMS 202, Fall 1997 Why Precision and Recall? n Get as much good stuff while at the same time getting as little junk as possible

20. Standard IR Evaluation n Precision n Recall Collection # relevant in collection # retrieved # relevant retrieved Retrieved Documents

21. Marti A. Hearst SIMS 202, Fall 1997 Precision/Recall Curves n There is a tradeoff between Precision and Recall n So measure Precision at different levels of Recall precision recall x x x x

22. Marti A. Hearst SIMS 202, Fall 1997 Precision/Recall Curves n Difficult to determine which of these two hypothetical results is better: precision recall x x x x

23. Precision/Recall Curves

24. Marti A. Hearst SIMS 202, Fall 1997 Document Cutoff Levels n Another way to evaluate: n Fix the number of documents retrieved at several levels: n top 5, top 10, top 20, top 50, top 100, top 500 n Measure precision at each of these levels n Take (weighted) average over results n This is a way to focus on high precision

25. Marti A. Hearst SIMS 202, Fall 1997 The E-Measure Combine Precision and Recall into one number (van Rijsbergen 79) P = precision R = recall b = measure of relative importance of P or R For example, b = 0.5 means user is twice as interested in precision as recall

26. Marti A. Hearst SIMS 202, Fall 1997 Expected Search Length n Documents are presented in order of predicted relevance n Search length: number of non-relevant documents that user must scan through in order to have their information need satisfied n The shorter the better n below: n=2, search length = 2; n=3, search length = 6 1 2 3 4 5 6 7 8 9 10 11 12 13 n y n y y y y n y n n n n

27. Marti A. Hearst SIMS 202, Fall 1997 What to Evaluate? n Effectiveness n Difficult to measure n Recall and Precision are one way n What might be others?

28. Marti A. Hearst SIMS 202, Fall 1997 How to Evaluate?

29. Marti A. Hearst SIMS 202, Fall 1997TREC n Text REtrieval Conference/Competition n Run by NIST (National Institute of Standards & Technology) n 1997 was the 6th year n Collection: 3 Gigabytes, >1 Million Docs n Newswire & full text news (AP, WSJ, Ziff) n Government documents (federal register) n Queries + Relevance Judgments n Queries devised and judged by “Information Specialists” n Relevance judgments done only for those documents retrieved -- not entire collection! n Competition n Various research and commercial groups compete n Results judged on precision and recall, going up to a recall level of 1000 documents

30. Marti A. Hearst SIMS 202, Fall 1997 Sample TREC queries (topics) Number: 168 Topic: Financing AMTRAK Description: A document will address the role of the Federal Government in financing the operation of the National Railroad Transportation Corporation (AMTRAK) Narrative: A relevant document must provide information on the government’s responsibility to make AMTRAK an economically viable entity. It could also discuss the privatization of AMTRAK as an alternative to continuing government subsidies. Documents comparing government subsidies given to air and bus transportation with those provided to aMTRAK would also be relevant.

31. Marti A. Hearst SIMS 202, Fall 1997 TREC n Benefits: n made research systems scale to large collections (pre-WWW) n allows for somewhat controlled comparisons n Drawbacks: n emphasis on high recall, which may be unrealistic for what most users want n very long queries, also unrealistic n comparisons still difficult to make, because systems are quite different on many dimensions n focus on batch ranking rather than interaction n no focus on the WWW

32. Marti A. Hearst SIMS 202, Fall 1997 TREC is changing n Emphasis on specialized “tracks” n Interactive track n Natural Language Processing (NLP) track n Multilingual tracks (Chinese, Spanish) n Filtering track n High-Precision n High-Performance n www-nlpir.nist.gov/TREC

33. Marti A. Hearst SIMS 202, Fall 1997 TREC Results n Differ each year n For the main track: n Best systems not statistically significantly different n Small differences sometimes have big effects n how good was the hyphenation model n how was document length taken into account n Systems were optimized for longer queries and all performed worse for shorter, more realistic queries n Excitement is in the new tracks

34. Marti A. Hearst SIMS 202, Fall 1997 Blair and Maron 1985 n A classic study of retrieval effectiveness n earlier studies were on unrealistically small collections n Studied an archive of documents for a legal suit n ~350,000 pages of text n 40 queries n focus on high recall n Used IBM’s STAIRS full-text system n Main Result: System retrieved less than 20% of the relevant documents for a particular information needs when lawyers thought they had 75% n But many queries had very high precision

35. Marti A. Hearst SIMS 202, Fall 1997 Blair and Maron, cont. n How they estimated recall n generated partially random samples of unseen documents n had users (unaware these were random) judge them for relevance n Other results: n two lawyers searches had similar performance n lawyers recall was not much different from paralegal’s

36. Marti A. Hearst SIMS 202, Fall 1997 Blair and Maron, cont. n Why recall was low n users can’t foresee exact words and phrases that will indicate relevant documents n “accident” referred to by those responsible as: “event,” “incident,” “situation,” “problem,” … n differing technical terminology n slang, misspellings n Perhaps the value of higher recall decreases as the number of relevant documents grows, so more detailed queries were not attempted once the users were satisfied

37. Marti A. Hearst SIMS 202, Fall 1997 Still to come: n Evaluating user interfaces