1. CS 430: Information Discovery
Lecture 2
Text Based Information Retrieval

2. Course Administration
Web site:
Notices:
See the home page of the course Web site
Sign-up sheet:
If you did not sign up at the first class, please sign up now.

3. Course Administration
Please send all questions about the course to:
The message will be sent to
William Arms
Pavel Dmitriev
Ariful Gani
Heng-Scheng Chuang

4. Course Administration
Discussion class, Wednesday, September 3
Upson B17, 7:30 to 8:30 p.m.
Prepare for the class as instructed on the course Web site.
Participation in the discussion classes is one third of the
grade, but tomorrow's class will not be included in the grade
calculation.
Due date of Assignment 1
This date may be changed. Watch the Notices on the Web site.

5. Functional View 1. Matching
Documents
Query
Index database
Mechanism for determining whether a document matches a query.
Set of hits

6. Matching: Recall and Precision
If information retrieval were perfect ...
Every hit would be relevant to the original query, and every relevant item in the body of information would be found.
Precision: percentage (or fraction) of the hits that are
relevant, i.e., the extent to which the set of hits
retrieved by a query satisfies the requirement that
generated the query.
Recall: percentage (or fraction) of the relevant items that are
found by the query, i.e., the extent to which the query
found all the items that satisfy the requirement.

7. Recall and Precision: Example
Collection of 10,000 documents, 50 on a specific topic
Ideal search finds these 50 documents and reject all others
Actual search identifies 25 documents; 20 are relevant but 5 were on other topics
Precision: 20/ 25 = (80% of hits were relevant)
Recall: 20/50 = 0.4 (40% of relevant were found)

8. Measuring Precision and Recall
Precision is easy to measure:
A knowledgeable person looks at each document that is identified and decides whether it is relevant.
In the example, only the 25 documents that are found need to be examined.
Recall is difficult to measure:
To know all relevant items, a knowledgeable person must go through the entire collection, looking at every object to decide if it fits the criteria.
In the example, all 10,000 documents must be examined.

9. Ranking Methods
Methods that look for matches assume that a document is either relevant to a query or not relevant.
Ranking methods: measure the degree of similarity between a query and a document.
Similar
Query
Documents
Similar: How similar is document to a request?

10. Functional View 2. Ranking Methods
Index database
Documents
Query
Mechanism for determining the similarity of the request representation to the information item representation.
Set of documents ranked by how similar they are to the query

11. Ranking: Recall and Precision
If information retrieval were perfect ...
Every document relevant to the original query would be ranked above every other document.
Precision and recall are functions of the rank order.
Precision(n): percentage (or fraction) of the n most highly
ranked documents that are relevant.
Recall (n) : percentage (or fraction) of the relevant items that
are in the n most highly ranked documents.

12. Precision and Recall with Ranking
Example
"Your query found 349,871 possibly relevant documents. Here are the first eight."
Examination of the first 8 finds that 5 of them are relevant.

13. Graph of Precision with Ranking
Relevant? Y N Y Y N Y N Y
Precision 1
1/ / / / / / / /8
Rank

14. Precision and Recall
Precision and recall measure the results of a single query using a specific search system applied to a specific set of documents.
Matching methods:
Precision and recall are single numbers.
Ranking methods:
Precision and recall are represented by functions (or graphs) of the rank order.

15. Text Based Information Retrieval
Most matching methods are based on Boolean operators.
Most ranking methods are based on the vector space model.
Many practical systems combine features of both approaches.
In the basic form, both approaches treat words as separate tokens with minimal attempt to interpret them linguistically.

16. Documents
A textual document is a digital object consisting of a sequence of words and other symbols, e.g., punctuation.
The individual words and other symbols are known as tokens or terms.
A textual document can be:
• Free text, also known as unstructured text, which is a
continuous sequence of tokens.
• Fielded text, also known as structured text, in which the text
is broken into sections that are distinguished by tags or other
markup.
[Methods of markup, e.g., XML, are covered in CS 431.]

17. Word Frequency Observation: Some words are more common than others.
Statistics: Most large collections of text documents have similar statistical characteristics. These statistics:
• influence the effectiveness and efficiency of data structures
used to index documents
• many retrieval models rely on them

18. Word Frequency Example The following example is taken from:
Jamie Callan, Characteristics of Text, 1997
Sample of 19 million words
The next slide shows the 50 commonest words in rank order (r), with their frequency (f).

19. f f f
the from or 54958
of he about
to million market
a year they
in its this
and be would
that was you
for company which 48273
is an bank
said has stock
it are trade
on have his
by but more
as will who
at say one
mr new their
with share

20. Rank Frequency Distribution
For all the words in a collection of documents, for each word w
f is the frequency that w appears
r is rank of w in order of frequency. (The most
commonly occurring word has rank 1, etc.) f
w has rank r and frequency f r

21. Rank Frequency Example
The next slide shows the words in Callan's data normalized. In this example:
r is the rank of word w in the sample.
f is the frequency of word w in the sample.
n is the total number of distinct words in the sample.

22. 1000*rf/n *rf/n *rf/n
the 59 from 92 or 101
of he about 102
to million market 101
a year they 103
in its this 105
and be would 107
that was you 106
for company which 107
is 72 an bank 109
said has stock 110
it are trade 112
on have his 114
by but more 114
as will who 106
at say one 107
mr new their 108
with share 114

23. Zipf's Law
If the words, w, in a collection are ranked, r, by their frequency, f, they roughly fit the relation:
r * f = c
Different collections have different constants c.
In English text, c tends to be about n / 10, where n is the number of distinct words in the collection.
For a weird but wonderful discussion of this and many other examples of naturally occurring rank frequency distributions, see:
Zipf, G. K., Human Behaviour and the Principle of Least Effort. Addison-Wesley, 1949

24. Luhn's Proposal
"It is here proposed that the frequency of word occurrence in an article furnishes a useful measurement of word significance. It is further proposed that the relative position within a sentence of words having given values of significance furnish a useful measurement for determining the significance of sentences. The significance factor of a sentence will therefore be based on a combination of these two measurements."
Luhn, H.P., The automatic creation of literature abstracts, IBM Journal of Research and Development, 2, (1958)

25. Cut-off Levels for Significance Words
Upper cut-off
Lower cut-off
Resolving power of significant words
Significant words r
from: Van Rijsbergen, Ch. 2

26. Methods that Build on Zipf's Law
Stop lists: Ignore the most frequent words (upper cut-off). Used by almost all systems.
Significant words: Ignore the most frequent and least frequent words (upper and lower cut-off). Rarely used.
Term weighting: Give differing weights to terms based on their frequency, with most frequent words weighed less. Used by almost all ranking methods.

27. Examples of Weighting Document frequency
A term that occurs in a few documents is likely to be a better discriminator that a term that appears in most or all documents.
Term frequency
A term that appears several times in a document is weighted more heavily than a term that appears only once.

28. Approaches to Weighting
Boolean information retrieval:
Weight of term i in document j:
w(i, j) = if term i occurs in document j
w(i, j) = otherwise
General weighting methods
0 < w(i, j) <= 1 if term i occurs in document j
(The use of weighting for ranking is the topic of Lecture 4.)