1. IR Theory: Evaluation Methods

2. Evaluation of IR Systems: Why?
Which is the best?
IR Model, Term Weighting, Indexing Method, User Interface, Search System
How to determine if a given approach is effective?
Is my new term weighting formula better than Okapi?
Is Google better than Naver?
Is VSM better than Boolean?
Without standard evaluation methods, assessment can be
Anecdotal
Biased
Incorrect
Search Engine

3. Evaluation of IR Systems: How?
Measure the degree to which information need is satisfied
Number and ranks of relevant documents in search results
User’s assessment
Usage
How often is the system used? Return rate
Change in user’s knowledge
Task completion
How to determine if a given approach is effective
Baseline system vs. Test system
Baseline system
Standard (top) system
Test system
Same as baseline except for the methods to test
Easy to assess if system differences are restricted
Multiple factors cause compound effects
e.g., IR model + term weight
Difficult to isolate the contribution of each factor
Search Engine

4. Evaluation of IR Systems: Challenges
What measure to use?
Effectiveness, Relevance, Utility, User Satisfaction
Who to judge?
Individual User, All Users, Subject Expert
How to measure?
Binary, Multi-level, Continuous
Difficulties of Evaluation
At the core, it is a subjective process
Information needs change over time and with learning
Evaluation criteria vary by user and context
Search Engine

5. IR Evaluation: Standard Approach
Measure to used
Effectiveness, Relevance, Utility, User Satisfaction
Who to judge?
Individual User, All Users, Subject Expert
How to measure?
Binary, Multi-level, Continuous
Process
Collect a set of documents (Collection)
Construct a set of queries (Queries/Topics)
For each query, human assessors identify relevant documents (Relevance Judgments)
IR systems use the queries against the collection to produce search results
Evaluate the results by comparing it to the relevance judgments
Search Engine

6. IR Evaluation: TREC Approach
Making relevance judgments for a large collection is practically impossible
#docs to review = #queries * #documents (collection size)
For 50 queries and 1 million documents, assessors need to review 50 million documents
Pooled Relevance Judgment
Create a pool of documents to review by collecting only top n (e.g., 100) documents from each system result
Justification
Previous research finding
Different systems retrieve different sets of documents
Assumption
Document pool created from top results will include majority of relevant documents in the collection
TREC finding
Pooled relevance does not affect relative rankings of systems
Search Engine

7. Evaluation Measures: Overview
Test Collection
Document Corpus
Topics: Description of information need
Relevance Judgments: Set of known relevant documents for each topic
Evaluation Measures
Precision (P): 정밀도
A measure of the system’s ability to present only the relevant items
P = number of relevant items retrieved / total number of items retrieved
Recall (R): 재현율
A measure of the system’s ability to retrieve all the relevant items
R = number of relevant items retrieved / number of relevant items in the collection
R-Precision (RP)
De-emphasizes the exact ranking of documents (good when many reldocs are retrieved)
RP = precision at rank R, where R= number of relevant items in the collection
Reciprocal Rank (RR)
Good for known item search
RR = 1/rank where first relevant item is retrieved
Search Engine

8. Evaluation Measures: Precision & Recall
Document Collection
Retrieved Documents
P = 
Relevant Documents
Retrieved Documents
Relevant Documents
Relevant documents retrieved
R = 
Search Engine

9. Evaluation Measures: Precision vs. Recall
Recall-Precision Tradeoff
Retrieves mostly relevant documents but misses many relevant documents
Optimum Performance 1
Precision 1
Recall
Retrieves most of relevant documents along with lots of junk
Search Engine

10. Evaluation Measures: Recall-Precision
Recall-Precision Graph
Precision at 11 standard recall levels
Commonly used to compare systems
closer to upper right  better system
Recall
Precision
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.6169
0.4517
0.3938
0.3243
0.2725
0.2224
0.1642
0.1342
0.0904
0.0472
0.0031
Search Engine

11. Evaluation Measures: Average Precision
Average Precision (AP)
Single valued measure that reflects performance over all relevant documents
Rewards the system that retrieves relevant documents at high ranks
Standard evaluation measure used to evaluate IR systems
Devised by TREC
Search Engine

12. Examples: Precision & Recall
rank
Doc#
Relevant?
Precision
Recall (Nrel=10) 1 42
Yes
1/1 =1.00
1/10 =0.10 2
221 No
1/2 =0.50 3
123
1/3 = 0.33 4 21
2/4 = 0.50
2/10 =0.20 5
111
3/5 = 0.60
3/10 =0.30 6 11
3/6 = 0.50 7 93
3/7 = 0.43 8
234
3/8 = 0.38 9
4/9 = 0.44
4/10 =0.40 10
254
4/10 = 0.40
333
5/11 = 0.45
5/10 =0.50 12
421
6/12 = 0.50
6/10 =0.60 13 45
6/13 = 0.46 14
761
7/14 = 0.50
7/10 =0.70 15
8/15 = 0.53
8/10 =0.80
Search Engine

13. Examples: Average Precision
rank
Doc#
Relevant?
Precision
Recall (Nrel=10) 1 42
Yes
1/1 =1.00
1/10 =0.10 2
221 No
1/2 =0.50 3
123
1/3 = 0.33 4 21
2/4 = 0.50
2/10 =0.20 5
111
3/5 = 0.60
3/10 =0.30 6 11
3/6 = 0.50 7 93
3/7 = 0.43 8
234
3/8 = 0.38 9
4/9 = 0.44
4/10 =0.40 10
254
4/10 = 0.40
333
5/11 = 0.45
5/10 =0.50 12
421
6/12 = 0.50
6/10 =0.60 13 45
6/13 = 0.46 14
761
7/14 = 0.50
7/10 =0.70 15
8/15 = 0.53
8/10 =0.80
Search Engine