1. CS 430: Information Discovery
Lecture 8
Evaluation of Retrieval Effectiveness 1

2. Course administration
Assignment 1 -- Report
The report is 33% of the grade. In particular, it should:
Describe the data structures used for the index file, postings, and document file, with brief explanation of why those structures were chosen.
Explain the mathematical expressions for the term weightings and how they were calculated.
If your report did not include this information, you
may resubmit the report before Thursday 5 p.m.
Instructions will be posted on the Web site.

3. Course administration
Discussion Class 4
Check the Web site.
(a) It is not necessary to study the entire paper in detail
(b) The PDF version of the file is damaged. Use the PostScript version.
Discussion Class 3
Because of the fire in Upson, this class will not count towards the final grade

4. Retrieval Effectiveness
Designing an information retrieval system has many decisions:
Manual or automatic indexing?
Natural language or controlled vocabulary?
What stoplists?
What stemming methods?
What query syntax?
etc.
How do we know which of these methods are most effective?
Is everything a matter of judgment?

5. Studies of Retrieval Effectiveness
• The Cranfield Experiments, Cyril W. Cleverdon, Cranfield
College of Aeronautics,
• SMART System, Gerald Salton, Cornell University,
• TREC, Donna Harman, National Institute of Standards and
Technology (NIST),

6. Cranfield Experiments (Example)
Comparative efficiency of indexing systems:
(Universal Decimal Classification, alphabetical subject index, a special facet classification, Uniterm system of co-ordinate indexing)
Four indexes prepared manually for each document in three batches of 6,000 documents -- total 18,000 documents, each indexed four times. The documents were reports and paper in aeronautics.
Indexes for testing were prepared on index cards and other cards.
Very careful control of indexing procedures.

7. Cranfield Experiments (continued)
Searching:
• 1,200 test questions, each satisfied by at least one document
• Reviewed by expert panel
• Searches carried out by 3 expert librarians
• Two rounds of searching to develop testing methodology
• Subsidiary experiments at English Electric Whetstone
Laboratory and Western Reserve University

8. The Cranfield Data
The Cranfield data was made widely available and used by other
researchers
• Salton used the Cranfield data with the SMART system (a) to study the relationship between recall and precision, and (b) to compare automatic indexing with human indexing
• Sparc Jones and van Rijsbergen used the Cranfield data for experiments in relevance weighting, clustering, definition of test corpora, etc.

9. Cranfield Experiments -- Measures of Effectiveness for Matching Methods
Cleverdon's work was applied to matching methods. He made extensive use of recall and precision, based on concept of relevance.
precision (%)
Each x represents one search. The graph illustrates the trade-off between precision and recall. x x x x x x x x x x
recall (%)

10. Typical precision-recall graph for different queries
Narrow, specific query
1.0
0.75
0.5
Broad, general query
0.25
recall
0.25
0.5
0.75
1.0

11. Some Cranfield Results
• The various manual indexing systems have similar retrieval efficiency
• Retrieval effectiveness using automatic indexing can be at
least as effective as manual indexing with controlled
vocabularies
-> original results from the Cranfield + SMART experiments
(published in 1967)
-> considered counter-intuitive
-> other results since then have supported this conclusion

12. Relevance Recall and precision: depend on concept of relevance
-> Is relevance a context-, task-independent property of documents?
"Relevance is the correspondence in context between an information requirement statement (a query) and an article (a document), that is, the extent to which the article covers the material that is appropriate to the requirement statement."
F. W. Lancaster, 1979

13. Relevance as a set comparison
D = set of documents
A = set of documents that satisfy some user-based criterion
B = set of documents identified by the search system

14. Measures based on relevance
retrieved relevant | A  B |
relevant | A |
retrieved | B |
retrieved not-relevant | B - A  B |
not-relevant | D - A |
recall = =
precision = =
fallout = =

15. Relevance
• Recall and precision values are for a specific set of documents and type of queries (e.g., subject-heading queries, title queries, paragraphs), and a specific information task.
• Relevance is subjective, but experimental evidence suggests that for textual documents different experts have similar judgments about relevance.
• Experiments have tried asking users to give a numeric relevance level or rank the relevant documents, but the results have been less consistent.
• Tests and judgment of relevance must use realistic queries.

16. Ranked retrieval: Recall and precision after retrieval of n documents
n relevant recall precision
1 yes
2 yes
3 no
4 yes
5 no
6 yes
7 no
8 no
9 no
10 no
11 no
12 no
13 yes
14 no
SMART system using Cranfield data, 200 documents in aeronautics of which 5 are relevant

17. Precision-recall graph
Note: Some authors plot recall against precision. 1 2
1.0 4
0.75 6 3 5
0.5 13 12
0.25
200
recall
0.25
0.5
0.75
1.0

18. 11 Point Precision (Recall Cut Off)
p(n) is precision at that point where recall has first reached n
Define 11 standard recall points p(r0), p(r1), ... p(r11),
where p(rn) = p(n/10)
Note: if p(rn) is not an exact data point, use interpolation

19. Recall cutoff graph: choice of interpolation points
precision 1 2
The blue line is the recall cutoff graph.
1.0 4
0.75 6 3 5
0.5 13 12
0.25
200
recall
0.25
0.5
0.75
1.0

20. Example: SMART System on Cranfield Data
Recall Precision
Precision values in blue are actual data.
Precision values in red are by interpolation.