1. Current Topics in Information Access: IR Background
Marti Hearst
Fall ‘98

2. Last Time The problem of information access
Matching task and search type
Why text is tough
Current topics and issues

3. Today Background on IR Basics System architecture Tokenization
Boolean queries
Term weighting and Ranking Algorithms
Inverted Indices
Evaluation

4. Some IR History
Roots in the scientific “Information Explosion” following WWII
Interest in computer-based IR from mid 1950’s
H.P. Luhn at IBM (1958)
Probabilistic models at Rand (Maron & Kuhns) (1960)
Boolean system development at Lockheed (‘60s)
Vector Space Model (Salton at Cornell 1965)
Statistical Weighting methods and theoretical advances (‘70s)
Refinements and Advances in application (‘80s)
User Interfaces, Large-scale testing and application (‘90s)

5. Information Retrieval
Task Statement
Build a system that retrieves documents that users are likely to find relevant to their information needs.

6. Structure of an IR System
Search
Line
Storage
Line
Interest profiles
& Queries
Documents
& data
Information Storage and Retrieval System
Rules of the game =
Rules for subject indexing +
Thesaurus (which consists of
Lead-In
Vocabulary
and
Indexing
Language
Formulating query in
terms of
descriptors
Indexing
(Descriptive and
Subject)
Storage of
profiles
Storage of
Documents
Store1: Profiles/
Search requests
Store2: Document
representations
Comparison/
Matching
Adapted from Soergel, p. 19
Potentially
Relevant
Documents

7. User’s
Information
Need
text input
Query
Parse

8. Collections
Pre-process
Index

9. User’s
Information
Need
Collections
Pre-process
text input
Query
Index
Parse
Rank or Match

10. User’s
Information
Need
Collections
Pre-process
text input
Query
Index
Parse
Rank or Match
Query Reformulation

11. Steps in a “typical IR System”
Document preprocessing
Tokenization
Stemming/Normalizing
Query processing
Interpretation of Syntax
(Query Expansion)
Retrieval of documents according to similarity to query
Presentation of Retrieval Results
Relevance Feedback/Query reformulation

12. Stemming and Morphological Analysis
Goal: “normalize” similar words
Morphology (“form” of words)
Inflectional Morphology
E.g,. inflect verb endings and noun number
Never change grammatical class
dog, dogs
Derivational Morphology
Derive one word from another,
Often change grammatical class
build, building; health, healthy

13. Automated Methods
Powerful multilingual tools exist for morphological analysis
PCKimmo, Xerox Lexical technology
Require a grammar and dictionary
Use “two-level” automata
Stemmers:
Very dumb rules work well (for English)
Porter Stemmer: Iteratively remove suffixes
Improvement: pass results through a lexicon
Use considered too expensive to store a dictionary

14. Errors Generated by Porter Stemmer (Krovetz 93)

15. Query Languages A way to express the question (information need)
Types:
Boolean
Natural Language
Stylized Natural Language
Form-Based (GUI)

16. Simple query language: Boolean
Terms + Connectors
terms
words
normalized (stemmed) words
phrases
thesaurus terms
connectors
AND OR
NOT

17. Boolean Queries Cat Cat OR Dog Cat AND Dog (Cat AND Dog)
(Cat AND Dog) OR Collar
(Cat AND Dog) OR (Collar AND Leash)
(Cat OR Dog) AND (Collar OR Leash)

18. Boolean Queries (Cat OR Dog) AND (Collar OR Leash)
Each of the following combinations works:
Cat x x x x x
Dog x x x x x
Collar x x x
Leash x x x x x

19. Boolean Queries (Cat OR Dog) AND (Collar OR Leash)
None of the following combinations work:
Cat x x
Dog x x
Collar x x
Leash x x

20. Boolean Logic B A

21. Boolean Searching Cracks Width Beams measurement Prestressed concrete
“Measurement of the
width of cracks in prestressed
concrete beams”
Formal Query:
cracks AND beams
AND Width_measurement
AND Prestressed_concrete
Cracks
Width
measurement
Beams
Relaxed Query:
(C AND B AND P) OR
(C AND B AND W) OR
(C AND W AND P) OR
(B AND W AND P)
Prestressed
concrete

22. Boolean Problems
Disjunctive (OR) queries lead to too many results, often off-target
Conjunctive (AND) queries lead to reduced, and commonly zero result
Not intuitive to most people

23. Advantages and Disadvantage of the Boolean Model
Disadvantages
Complete expressiveness for any identifiable subset of collection
Exact and simple to program
The whole panoply of Boolean Algebra available
Complex query syntax is often misunderstood (if understood at all)
Problems of Null output and Information Overload
Output is not ordered in any useful fashion

24. Psuedo-Boolean Queries
A new notation, from web search
+cat dog +collar leash
Does not mean the same thing!
Need a way to group combinations.
Phrases:
“stray cat” AND “frayed collar”
+“stray cat” + “frayed collar”

25. Boolean Extensions Fuzzy Logic Adds weights to each term/concept
ta AND tb is interpreted as MIN(w(ta),w(tb))
ta OR tb is interpreted as MAX (w(ta),w(tb))
Proximity/Adjacency operators
Interpreted as additional constraints on Boolean AND

26. Ranking Algorithms Assign weights to the terms in the query.
Assign weights to the terms in the documents.
Compare the weighted query terms to the weighted document terms.
Rank order the results.

27. User’s
Information
Need
Collections
Pre-process
text input
Query
Index
Parse
Rank or Match

28. Indexing and Representation: The Vector Space Model
Document represented by a vector of terms
Words (or word stems)
Phrases (e.g. computer science)
Removes words on “stop list”
Documents aren’t about “the”
Often assumed that terms are uncorrelated.
Correlations between term vectors implies a similarity between documents.
For efficiency, an inverted index of terms is often stored.

29. Document Representation What values to use for terms
Boolean (term present /absent)
tf (term frequency) - Count of times term occurs in document.
The more times a term t occurs in document d the more likely it is that t is relevant to the document.
Used alone, favors common words, long documents.
df document frequency
The more a term t occurs throughout all documents, the more poorly t discriminates between documents
tf-idf term frequency * inverse document frequency -
High value indicates that the word occurs more often in this document than average.

30. Document Vectors Documents are represented as “bags of words”
Represented as vectors when used computationally
A vector is like an array of floating point
Has direction and magnitude
Each vector holds a place for every term in the collection
Therefore, most vectors are sparse

31. Vector Representation
Documents and Queries are represented as vectors.
Position 1 corresponds to term 1, position 2 to term 2, position t to term t

32. Document Vectors A B C D E F G H I
Document ids A B C D E F G H I
nova galaxy heat h’wood film role diet fur

33. Assigning Weights Want to weight terms highly if they are
frequent in relevant documents … BUT
infrequent in the collection as a whole

34. Assigning Weights tf x idf measure: term frequency (tf)
inverse document frequency (idf)

35. tf x idf
Normalize the term weights (so longer documents are not unfairly given more weight)

36. tf x idf normalization
Normalize the term weights (so longer documents are not unfairly given more weight)
normalize usually means force all values to fall within a certain range, usually between 0 and 1, inclusive.

37. Vector Space Similarity Measure combine tf x idf into a similarity measure

38. Computing Similarity Scores
1.0
0.8
0.6
0.4
0.2
0.2
0.4
0.6
0.8
1.0

39. Documents in Vector Space

40. Computing a similarity score

41. Similarity Measures Simple matching (coordination level match)
Dice’s Coefficient
Jaccard’s Coefficient
Cosine Coefficient
Overlap Coefficient

42. Problems with Vector Space
There is no real theoretical basis for the assumption of a term space
it is more for visualization that having any real basis
most similarity measures work about the same regardless of model
Terms are not really orthogonal dimensions
Terms are not independent of all other terms

43. Probabilistic Models
Rigorous formal model attempts to predict the probability that a given document will be relevant to a given query
Ranks retrieved documents according to this probability of relevance (Probability Ranking Principle)
Relies on accurate estimates of probabilities for accurate results

44. Probabilistic Retrieval
Goes back to 1960’s (Maron and Kuhns)
Robertson’s “Probabilistic Ranking Principle”
Retrieved documents should be ranked in decreasing probability that they are relevant to the user’s query.
How to estimate these probabilities?
Several methods (Model 1, Model 2, Model 3) with different emphases on how estimates are done.

45. Probabilistic Models: Some Notation
D = All present and future documents
Q = All present and future queries
(Di,Qj) = A document query pair
x = class of similar documents,
y = class of similar queries,
Relevance is a relation:

46. Probabilistic Models: Logistic Regression
Probability of relevance is based on Logistic regression from a sample set of documents to determine values of the coefficients. At retrieval the probability estimate is obtained by:
For the 6 X attribute measures shown next

47. Probabilistic Models: Logistic Regression attributes
Average Absolute Query Frequency
Query Length
Average Absolute Document Frequency
Document Length
Average Inverse Document Frequency
Inverse Document Frequency
Number of Terms in common between query and document -- logged

48. Probabilistic Models: Logistic Regression
Estimates for relevance based on log-linear model with various statistical measures of document content as independent variables.
Log odds of relevance is a linear function of attributes:
Term contributions summed:
Probability of Relevance is inverse of log odds:

49. Probabilistic Models Advantages Disadvantages Strong theoretical basis
In principle should supply the best predictions of relevance given available information
Can be implemented similarly to Vector
Relevance information is required -- or is “guestimated”
Important indicators of relevance may not be term -- though terms only are usually used
Optimally requires on-going collection of relevance information

50. Vector and Probabilistic Models
Support “natural language” queries
Treat documents and queries the same
Support relevance feedback searching
Support ranked retrieval
Differ primarily in theoretical basis and in how the ranking is calculated
Vector assumes relevance
Probabilistic relies on relevance judgments or estimates

51. Simple Presentation of Results
Order by similarity
Decreased order of presumed relevance
Items retrieved early in search may help generate feedback by relevance feedback
Select top k documents
Select documents within of query

52. Problems with Vector Space
There is no real theoretical basis for the assumption of a term space
it is more for visualization that having any real basis
most similarity measures work about the same regardless of model
Terms are not really orthogonal dimensions
Terms are not independent of all other terms

53. Evaluation Relevance Evaluation of IR Systems Precision vs. Recall
Cutoff Points
Test Collections/TREC
Blair & Maron Study

54. What to Evaluate? How much learned about the collection?
How much learned about a topic?
How much of the information need is satisfied?
How inviting the system is?

55. What to Evaluate?
What can be measured that reflects users’ ability to use system? (Cleverdon 66)
Coverage of Information
Form of Presentation
Effort required/Ease of Use
Time and Space Efficiency
Recall
proportion of relevant material actually retrieved
Precision
proportion of retrieved material actually relevant
effectiveness

56. Relevance In what ways can a document be relevant to a query?
Answer precise question precisely.
Partially answer question.
Suggest a source for more information.
Give background information.
Remind the user of other knowledge.
Others ...

57. Standard IR Evaluation
Retrieved
Documents
Precision
Recall
# relevant retrieved
# retrieved
# relevant retrieved
# relevant in collection
Collection

58. Precision/Recall Curves
There is a tradeoff between Precision and Recall
So measure Precision at different levels of Recall
precision x x x x
recall

59. Precision/Recall Curves
Difficult to determine which of these two hypothetical results is better:
precision x x x x
recall

60. Precision/Recall Curves

61. Document Cutoff Levels
Another way to evaluate:
Fix the number of documents retrieved at several levels:
top 5, top 10, top 20, top 50, top 100, top 500
Measure precision at each of these levels
Take (weighted) average over results
This is a way to focus on high precision

62. The E-Measure
Combine Precision and Recall into one number (van Rijsbergen 79)
P = precision
R = recall
b = measure of relative importance of P or R
For example,
b = 0.5 means user is twice as interested in
precision as recall

63. TREC Text REtrieval Conference/Competition
Run by NIST (National Institute of Standards & Technology)
1997 was the 6th year
Collection: 3 Gigabytes, >1 Million Docs
Newswire & full text news (AP, WSJ, Ziff)
Government documents (federal register)
Queries + Relevance Judgments
Queries devised and judged by “Information Specialists”
Relevance judgments done only for those documents retrieved -- not entire collection!
Competition
Various research and commercial groups compete
Results judged on precision and recall, going up to a recall level of 1000 documents

64. Sample TREC queries (topics)
<num> Number: 168
<title> Topic: Financing AMTRAK
<desc> Description:
A document will address the role of the Federal Government in financing the operation of the National Railroad Transportation Corporation (AMTRAK)
<narr> Narrative: A relevant document must provide information on the government’s responsibility to make AMTRAK an economically viable entity. It could also discuss the privatization of AMTRAK as an alternative to continuing government subsidies. Documents comparing government subsidies given to air and bus transportation with those provided to aMTRAK would also be relevant.

65. TREC
Benefits:
made research systems scale to large collections (pre-WWW)
allows for somewhat controlled comparisons
Drawbacks:
emphasis on high recall, which may be unrealistic for what most users want
very long queries, also unrealistic
comparisons still difficult to make, because systems are quite different on many dimensions
focus on batch ranking rather than interaction
no focus on the WWW

66. TREC Results Excitement is in the new tracks Interactive Multilingual
Differ each year
For the main track:
Best systems not statistically significantly different
Small differences sometimes have big effects
how good was the hyphenation model
how was document length taken into account
Systems were optimized for longer queries and all performed worse for shorter, more realistic queries
Excitement is in the new tracks
Interactive
Multilingual
NLP

67. Blair and Maron 1985 Highly influential paper
A classic study of retrieval effectiveness
earlier studies were on unrealistically small collections
Studied an archive of documents for a legal suit
~350,000 pages of text
40 queries
focus on high recall
Used IBM’s STAIRS full-text system
Main Result: System retrieved less than 20% of the relevant documents for a particular information needs when lawyers thought they had 75%
But many queries had very high precision

68. Blair and Maron, cont. Why recall was low
users can’t foresee exact words and phrases that will indicate relevant documents
“accident” referred to by those responsible as:
“event,” “incident,” “situation,” “problem,” …
differing technical terminology
slang, misspellings
Perhaps the value of higher recall decreases as the number of relevant documents grows, so more detailed queries were not attempted once the users were satisfied

69. Blair and Maron, cont. Why recall was low
users can’t foresee exact words and phrases that will indicate relevant documents
“accident” referred to by those responsible as:
“event,” “incident,” “situation,” “problem,” …
differing technical terminology
slang, misspellings
Perhaps the value of higher recall decreases as the number of relevant documents grows, so more detailed queries were not attempted once the users were satisfied

70. Information
need
Collections
Pre-process
text input
Query
Index
Parse
Rank or Match

71. Creating a Keyword Index
For each document
Tokenize the document
Break it up into tokens: words, stems, punctuation
There are many variations on this
Record which tokens occurred in this document
Called an Inverted Index
Dictionary: a record of all the tokens in the collection and their overall frequency
Postings File: a list recording for each token, which document it occurs in and how often it occurs

72. Inverted files Permit fast search for individual terms
Search results for each term is a list of document IDs (and optionally, frequency and/or positional information)
These lists can be used to solve Boolean queries:
country: d1, d2
manor: d2
country and manor: d2

73. Inverted Files Lots of alternative implementations
E.g.: Cheshire builds within-document frequency using a hash table during parsing
Document IDs and frequency info are stored in a B-tree index keyed by the term.

74. How Are Inverted Files Created
Documents are parsed to extract tokens. These are saved with the Document ID.
Doc 1
Doc 2
Now is the time
for all good men
to come to the aid
of their country
It was a dark and
stormy night in
the country
manor. The time
was past midnight

75. How Inverted Files are Created
After all documents have been parsed the inverted file is sorted

76. How Inverted Files are Created
Multiple term entries for a single document are merged and frequency information added

77. How Inverted Files are Created
Then the file is split into a Dictionary and a Postings file

78. An Example IR System CHESHIRE system (Ray Larson, SIMS)
Full Service Full Text Search
Client/Server architecture
Z39.50 IR protocol
Interprets documents written in SGML
Probabilistic Ranking

79. Next Time
Background on User Interfaces and Informaiton Access