1. Evaluation of Information Retrieval Systems Xiangming Mu

2. The Cranfield Model Based on a series of studies conducted at Cranfield, England The principal components included –A test collection of documents –A set of queries –A set of relevance judgments Binary assessments of the documents against the queries (relevant or not relevant)

3. The Cranfield Model (cont’) Document set –Relevant and retrieved documents----a –Nonrelevant and retrieved documents---b –Relevant and nonretrieved documents----c –Nonrelevant and nonretrieved documents---d Recall and precision –Recall = a/(a+c) –Precision= a/(a+b) –Recall is the proportion of relevant documents that are retrieved, and measures the completeness of the output –Precision is the proportion of retrieved documents that are relevant, and assess the accuracy of the output RetrievedNon-retrieved Related ac Non-related bd

4. Recall and Precision An inverse relationship between recall and precision A measure of effectiveness--- the combination of recall and precision –It is easy to get a high recall or a high precision Other methods –Expected Search Length (ESL)– the number of unwanted documents the user can expect to examine before finding the desired number of relevant documents

5. Problems The measurement of recall is very difficult and expensive –as a result, relative recall is introduced Cranfield model is based on human judgment ( relevant or not), and is a binary value (yes/no) Do not support interactions between users and systems---a batch approach

6. Problems (cont’) User satisfaction or system effectiveness? –Why and when users stopped (psychological, logistics, system interface, etc.)? –How users form their queries (user behaviors)? –What is the users’ tasks? Ignored human demographic differences Ignored users’ information need, or assumes that it is static Assumptions –Queries/users/information needs represent a larger population –Documents represent a larger collection of various type documents

7. Users, queries, and relevance IR Goal, Intention Task, Problem Cognitive Need Query Objects in Query Results

8. Projects STAIRS project –350,000 pages of online information of legal materials –Conducted on IBM’s STAIRS system TREC experiments

9. Text Retrieval Conference (TREC) Coordinated by the National Institute of Standards and Technology (NIST) Sponsored by the Advanced Research Projects Agency (ARPA) of the Department of Defense. First TREC proceedings were published in 1993

10. TREC (cont’) The goals –Encourage research in text retrieval based on large-scale test collections –Increase communication among researchers in industry, academia, and government –Speed the transfer of technology from research laboratories into commercial products –Increase the availability of appropriate evaluation techniques

11. TREC (cont’) Tasks –Routing (or filtering, profiling) tasks assume that the same questions are asked but that new documents are continually being searched. –Ad hoc tasks consist of new questions continually being posed to a static set of data Documents –3 gigabytes documents vary in number and length

12. TREC (cont’) Relevance –Created using the pooling method--- relevant documents were generated by taking the top x(100 or 200) documents retrieved by each system for a particular topic and merging them –Documents not retrieved in the top x by any of the systems were assumed to be Nonrelevant –Relevance was measured on a binary scale

13. Internet Search Engines How it works –Using autonomous search robots or spiders –Moving from one URL to another –Extracting data of interest and add to their indexes as they move Issues –The dynamic nature of the Internet search engine database –How to deal with system/user interaction –The existence of duplicate records –Recall is hard to evaluate