1. Information Retrieval (3)
Prof. Dragomir R. Radev

2. 5. Evaluation of IR systems
SI650 Winter 2010 …
5. Evaluation of IR systems
Reference collections
TREC

3. Relevance Difficult to change: fuzzy, inconsistent
Methods: exhaustive, sampling, pooling, search-based

4. Contingency table retrieved not retrieved relevant w=tp x=fn
n1 = w + x
not relevant
y=fp
z=tn N
n2 = w + y

5. Precision and Recall w
Recall:
w+x w
Precision:
w+y

6. Exercise
Go to Google ( and search for documents on Tolkien’s “Lord of the Rings”. Try different ways of phrasing the query: e.g., Tolkien, “JRR Tolkien”, +”JRR Tolkien” +Lord of the Rings”, etc. For each query, compute the precision (P) based on the first 10 documents returned by AltaVista.
Note! Before starting the exercise, have a clear idea of what a relevant document for your query should look like. Try different information needs.
Later, try different queries.

7. [From Salton’s book]

9. Interpolated average precision (e.g., 11pt)
Interpolation – what is precision at recall=0.5?

10. Issues Why not use accuracy A=(w+z)/N? Average precision
Average P at given “document cutoff values”
Report when P=R
F measure: F=(b2+1)PR/(b2P+R)
F1 measure: F1 = 2/(1/R+1/P) : harmonic mean of P and R

11. Kappa N: number of items (index i) n: number of categories (index j)
k: number of annotators

12. Kappa example
J1+
J1-
TOTAL
J2+
300 10
310
J2- 20 70 90
320 80
400

13. Kappa (cont’d) P(A) = 370/400 = 0.925
P (E) = * * = 0.665
K = ( )/( ) = 0.776
Kappa higher than 0.67 is tentatively acceptable; higher than 0.8 is good

14. Sample TREC query <top> <num> Number: 305
<title> Most Dangerous Vehicles
<desc> Description:
Which are the most crashworthy, and least crashworthy,
passenger vehicles?
<narr> Narrative:
A relevant document will contain information on the crashworthiness of a given vehicle or vehicles that can be used to draw a comparison with other vehicles. The document will have to describe/compare vehicles, not drivers. For instance, it should be expected that vehicles preferred by year-olds would be involved in more crashes, because that age group is involved in more crashes. I would view number of fatalities per 100 crashes to be more revealing of a vehicle's crashworthiness than the number of crashes per 100,000 miles, for example.
</top> LA FT LA LA LA LA FT LA FT LA LA FT LA
LA LA LA LA LA LA LA LA LA FT LA LA

15. <DOCNO> LA031689-0177 </DOCNO>
<DOCID> </DOCID>
<DATE><P>March 16, 1989, Thursday, Home Edition </P></DATE>
<SECTION><P>Business; Part 4; Page 1; Column 5; Financial Desk </P></SECTION>
<LENGTH><P>586 words </P></LENGTH>
<HEADLINE><P>AGENCY TO LAUNCH STUDY OF FORD BRONCO II AFTER HIGH RATE OF ROLL-OVER ACCIDENTS </P></HEADLINE>
<BYLINE><P>By LINDA WILLIAMS, Times Staff Writer </P></BYLINE>
<TEXT>
<P>The federal government's highway safety watchdog said Wednesday that the Ford Bronco II appears to be involved in more fatal roll-over
accidents than other vehicles in its class and that it will seek to determine if the vehicle itself contributes to the accidents. </P>
<P>The decision to do an engineering analysis of the Ford Motor Co. utility-sport vehicle grew out of a federal accident study of the
Suzuki Samurai, said Tim Hurd, a spokesman for the National Highway Traffic Safety Administration. NHTSA looked at Samurai accidents after
Consumer Reports magazine charged that the vehicle had basic design flaws. </P>
<P>Several Fatalities </P>
<P>However, the accident study showed that the "Ford Bronco II appears to have a higher number of single-vehicle, first event roll-overs,
particularly those involving fatalities," Hurd said. The engineering analysis of the Bronco, the second of three levels of investigation
conducted by NHTSA, will cover the Bronco II models, the agency said. </P>
<P>According to a Fatal Accident Reporting System study included in the September report on the Samurai, 43 Bronco II single-vehicle
roll-overs caused fatalities, or 19 of every 100,000 vehicles. There were eight Samurai fatal roll-overs, or 6 per 100,000; 13 involving
the Chevrolet S10 Blazers or GMC Jimmy, or 6 per 100,000, and six fatal Jeep Cherokee roll-overs, for 2.5 per 100,000. After the
accident report, NHTSA declined to investigate the Samurai. </P>
...
</TEXT>
<GRAPHIC><P> Photo, The Ford Bronco II "appears to have a higher
number of single-vehicle, first event roll-overs," a federal official
said. </P></GRAPHIC>
<SUBJECT>
<P>TRAFFIC ACCIDENTS; FORD MOTOR CORP; NATIONAL HIGHWAY TRAFFIC SAFETY ADMINISTRATION; VEHICLE INSPECTIONS;
RECREATIONAL VEHICLES; SUZUKI MOTOR CO; AUTOMOBILE SAFETY </P>
</SUBJECT>
</DOC>

16. TREC (cont’d) http://trec.nist.gov/tracks.html

17. Most used reference collections
Generic retrieval: OHSUMED, CRANFIELD, CACM
Text classification: Reuters, 20newsgroups
Question answering: TREC-QA
Web: DOTGOV, wt100g
Blogs: Buzzmetrics datasets
TREC ad hoc collections, 2-6 GB
TREC Web collections, 2-100GB

18. Comparing two systems Comparing A and B One query?
Average performance?
Need: A to consistently outperform B
[this slide: courtesy James Allan]

19. The sign test Example 1: Example 2: A > B (12 times)
p < (significant at the 5% level)
Example 2:
A > B (18 times)
A < B (9 times)
p < (not significant at the 5% level)
[this slide: courtesy James Allan]

20. Other tests
Student t-test: takes into account the actual performances, not just which system is better
Wilcoxon Matched-Pairs Signed-Ranks Test

21. 6. Automated indexing/labeling
IR Winter 2010 …
6. Automated indexing/labeling
Compression

22. Indexing methods
Manual: e.g., Library of Congress subject headings, MeSH
Automatic: e.g., TF*IDF based

23. LOC subject headings http://www.loc.gov/catdir/cpso/lcco/lcco.html
A -- GENERAL WORKS B -- PHILOSOPHY. PSYCHOLOGY. RELIGION C -- AUXILIARY SCIENCES OF HISTORY D -- HISTORY (GENERAL) AND HISTORY OF EUROPE E -- HISTORY: AMERICA F -- HISTORY: AMERICA G -- GEOGRAPHY. ANTHROPOLOGY. RECREATION H -- SOCIAL SCIENCES J -- POLITICAL SCIENCE K -- LAW L -- EDUCATION M -- MUSIC AND BOOKS ON MUSIC N -- FINE ARTS P -- LANGUAGE AND LITERATURE Q -- SCIENCE R -- MEDICINE S -- AGRICULTURE T -- TECHNOLOGY U -- MILITARY SCIENCE V -- NAVAL SCIENCE Z -- BIBLIOGRAPHY. LIBRARY SCIENCE. INFORMATION RESOURCES (GENERAL)

24. Medicine CLASS R - MEDICINE Subclass R R5-920 Medicine (General)
R General works
R History of medicine. Medical expeditions
R Medicine as a profession. Physicians
R Medicine and the humanities. Medicine and disease in relation to history, literature, etc.
R Directories
R Missionary medicine. Medical missionaries
R Medical philosophy. Medical ethics
R Medicine and disease in relation to psychology. Terminal care. Dying
R Medical personnel and the public. Physician and the public
R Practice of medicine. Medical practice economics
R Medical education. Medical schools. Research
R Medical technology
R Biomedical engineering. Electronics. Instrumentation
R Computer applications to medicine. Medical informatics
R864 Medical records
R Medical physics. Medical radiology. Nuclear medicine

25. Automatic methods TF*IDF: pick terms with the highest TF*IDF scores
Centroid-based: pick terms that appear in the centroid with high scores
The maximal marginal relevance principle (MMR)
Related to summarization, snippet generation

26. Compression
Methods
Fixed length codes
Huffman coding
Ziv-Lempel codes

27. Fixed length codes Binary representations ASCII
Representational power (2k symbols where k is the number of bits)

28. Variable length codes Alphabet: A .- N -. 0 ----- B -... O --- 1 .----
C -.-.  P .--. 
D -..  Q --.-  —
E .  R
F S
G --. T - 
H U ..- 
I ..  V ...- 
J .---  W .-- 
K -.-  X -..-
L .-..  Y -.—
M --  Z --..
Demo:

29. Most frequent letters in English
E T A O I N S H R D L U
Demo:
Also: bigrams:
TH HE IN ER AN RE ND AT ON NT

30. Huffman coding Developed by David Huffman (1952)
Average of 5 bits per character (37.5% compression)
Based on frequency distributions of symbols
Algorithm: iteratively build a tree of symbols starting with the two least frequent symbols

32. 1 1 1 g 1 1 1 i j f c 1 1 b d a 1 e h

34. Exercise
Consider the bit string:
Use the Huffman code from the example to decode it.
Try inserting, deleting, and switching some bits at random locations and try decoding.

35. Extensions
Word-based
Domain/genre dependent models

36. Ziv-Lempel coding Two types - one is known as LZ77 (used in GZIP)
Code: set of triples <a,b,c>
a: how far back in the decoded text to look for the upcoming text segment
b: how many characters to copy
c: new character to add to complete segment

37. <0,0,p> p
<0,0,e> pe
<0,0,t> pet
<2,1,r> peter
<0,0,_> peter_
<6,1,i> peter_pi
<8,2,r> peter_piper
<6,3,c> peter_piper_pic
<0,0,k> peter_piper_pick
<7,1,d> peter_piper_picked
<7,1,a> peter_piper_picked_a
<9,2,e> peter_piper_picked_a_pe
<9,2,_> peter_piper_picked_a_peck_
<0,0,o> peter_piper_picked_a_peck_o
<0,0,f> peter_piper_picked_a_peck_of
<17,5,l> peter_piper_picked_a_peck_of_pickl
<12,1,d> peter_piper_picked_a_peck_of_pickled
<16,3,p> peter_piper_picked_a_peck_of_pickled_pep
<3,2,r> peter_piper_picked_a_peck_of_pickled_pepper
<0,0,s> peter_piper_picked_a_peck_of_pickled_peppers

38. Links on text compression
Data compression:
Calgary corpus:
Huffman coding: LZ

39. 100 alternative search engines

40. Readings
2: MRS9
3: MRS13, MRS14
4: MRS15, MRS16