1. 1 CS 430 / INFO 430 Information Retrieval Lecture 3 Searching Full Text 3

2. 2 Course Administration

3. 3 Books on Information Retrieval Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Modern Information Retrieval, Addison Wesley, 1999. Covers most of the standard topics. Chapters vary in quality. William B. Frakes and Ricardo Baeza-Yates, Information Retrieval Data Structures and Algorithms. Prentice Hall, 1992. Good coverage of algorithms, but out of date. Several good chapters. G. Salton and M. J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, 1983. The classic description of the underlying methods.

4. 4 Books on Information Retrieval Amy Langville and Carl Meyer, Google's PageRank and Beyond: the Science of Search Engine Rankings. Princeton University Press, 2006. A mathematical treatment of link based methods with summaries for the non-mathematician.

5. 5 Observations from Discussion Class 1 The quality of a search service can be no better than the quality of information in the underlying document collection. The interface presented to a trained user can offer more features than the interface presented to an untrained user. Very different methods are needed for searching a full text index of online materials (which are immediately available) and a catalog of physical items (which must be retrieved). Subject headings are a form of controlled vocabulary that can be very effective in specialized domains, e.g., medicine.

6. 6 From the Library of Congress FAQ 13. How does relevance ranking work and where is it available? The results of a Keyword search on the "Basic Search" screen are displayed in order of their relevance to your search. In the Title List following a Keyword search, the display will include a column with a "Relevance" indicator; the most relevant items display first. The relevance is determined by three factors: * Uniqueness of search terms within the database. * Proximity of search terms to each other within the catalog record. * Number of different search terms present in a catalog record.

7. 7 Information Retrieval Family Trees Gerald Salton Cornell Donna Harman NIST Michael Lesk Bell Labs, etc. Karen Sparck Jones Cambridge Keith Van Rijsbergen Glasgow Bruce Croft University of Massachusetts, Amherst Cyril Cleverdon Cranfield

8. 8 Similarity Ranking Methods Query DocumentsIndex database Mechanism for determining the similarity of the query to the document. Set of documents ranked by how similar they are to the query

9. 9 Two Documents Represented in 3-Dimensional Term Vector Space t1t1 t2t2 t3t3 d1d1 d2d2 

10. 10 Vector Space Revision x = (x 1, x 2, x 3,..., x n ) is a vector in an n-dimensional vector space Length of x is given by (extension of Pythagoras's theorem) |x| 2 = x 1 2 + x 2 2 + x 3 2 +... + x n 2 If x 1 and x 2 are vectors: Inner product (or dot product) is given by x 1.x 2 = x 11 x 21 + x 12 x 22 + x 13 x 23 +... + x 1n x 2n Cosine of the angle between the vectors x 1 and x 2: cos (  ) = x 1.x 2 |x 1 | |x 2 |

11. 11 Weighting: Unnormalized Form of Term Frequency (tf) ant bee cat dog eel fox gnu hog length d 1 2 1  5 d 2 1 1 4 1  19 d 3 1 1 1 1 1  5 Weights: t ij = frequency that term j occurs in document i documenttextterms d 1 ant ant bee ant bee d 2 dog bee dog hog dog ant dogant bee dog hog d 3 cat gnu dog eel fox cat dog eel fox gnu

12. 12 Example (continued) d 1 d 2 d 3 d 1 10.31 0 d 2 0.31 10.41 d 3 00.41 1 Similarity of documents in example: Similarity depends upon the weights given to the terms. [Note differences in results from example in Lecture 2.]

13. 13 Similarity: Weighting by Unnormalized Form of Term Frequency ant bee cat dog eel fox gnu hog length q 1 1 √2 d 1 2 1  5 d 2 1 1 4 1  19 d 3 1 1 1 1 1  5 query qant dog documenttextterms d 1 ant ant bee ant bee d 2 dog bee dog hog dog ant dogant bee dog hog d 3 cat gnu dog eel fox cat dog eel fox gnu

14. 14 Calculate Ranking d 1 d 2 d 3 q 2/√10 5/√38 1/√10 0.63 0.81 0.32 Similarity of query to documents in example: If the query q is searched against this document set, the ranked results are: d 2, d 1, d 3

15. 15 Choice of Weights ant bee cat dog eel fox gnu hog q ? ? d 1 ? ? d 2 ? ? ? ? d 3 ? ? ? ? ? query qant dog documenttextterms d 1 ant ant bee ant bee d 2 dog bee dog hog dog ant dogant bee dog hog d 3 cat gnu dog eel fox cat dog eel fox gnu What weights lead to the best information retrieval?

16. 16 Evaluation Before we can decide whether one system of weights is better than another, we need systematic and repeatable methods to evaluate methods of information retrieval. Evaluation is the topic of Lecture 8 and 9.

17. 17 Methods for Selecting Weights Empirical Test a large number of possible weighting schemes with actual data. (This lecture, based on work of Salton, et al.) Model based Develop a mathematical model of word distribution and derive weighting scheme theoretically. (Probabilistic model of information retrieval.)

18. 18 Weighting Term Frequency (tf) Suppose term j appears f ij times in document i. What weighting should be given to a term j? Term Frequency: Concept A term that appears many times within a document is likely to be more important than a term that appears only once.

19. 19 Normalized Form of Term Frequency: Free-text Document Length of document Unnormalized method is to use f ij as the term frequency....but, in free-text documents, terms are likely to appear more often in long documents. Therefore f ij should be scaled by some variable related to document length.

20. 20 Term Frequency: Free-text Document A standard method for free-text documents Scale f ij relative to the frequency of other terms in the document. This partially corrects for variations in the length of the documents. Let m i = max (f ij ) i.e., m i is the maximum frequency of any term in document i. Term frequency (tf): tf ij = f ij / m i Note: There is no special justification for taking this form of term frequency except that it works well in practice and is easy to calculate. i

21. 21 Weighting Inverse Document Frequency (idf) Suppose term j appears f ij times in document i. What weighting should be given to a term j? Inverse Document Frequency: Concept By Zipf's Law we know that some terms appear much more often than others across the documents of a corpus. A term that occurs in only a few documents is likely to be a better discriminator that a term that appears in most or all documents.

22. 22 Inverse Document Frequency Suppose there are n documents and that the number of documents in which term j occurs is n j. Simple method We could define document frequency as n j /n. A possible method might be to use the inverse, n/n j, as a weight. This would give greater weight to words that appear in fewer documents.

23. 23 Inverse Document Frequency A standard method The simple method over-emphasizes small differences. Therefore use a logarithm. Inverse document frequency (idf): idf j = log 2 (n/n j ) + 1 n j > 0 Note: There is no special justification for taking this form of inverse document frequency except that it works well in practice and is easy to calculate. [The rationale behind this choice of weight is discussed in Discussion Class 3.]

24. 24 Example of Inverse Document Frequency Example n = 1,000 documents term jn j n/n j idf j A10010.004.32 B5002.002.00 C9001.111.13 D1,0001.001.00 From: Salton and McGill

25. 25 Full Weighting: A Standard Form of tf.idf Practical experience has demonstrated that weights of the following form perform well in a wide variety of circumstances: (weight of term j in document i) = (term frequency) * (inverse document frequency) A standard tf.idf weighting scheme, for free text documents, is: t ij = tf ij * idf j = (f ij / m i ) * (log 2 (n/n j ) + 1) when n j > 0

26. 26 Vector Similarity Computation with Weights Documents in a collection are assigned terms from a set of n terms The term vector space W is defined as: if term k does not occur in document d i, w ik = 0 if term k occurs in document d i, w ik is greater than zero (w ik is called the weight of term k in document d i ) Similarity between d i and d j is defined as:  w ik w jk |d i | |d j | Where d i and d j are the corresponding weighted term vectors k=1 n cos(d i, d j ) =

27. 27 Structured Text Structured text Structured texts, e.g., queries, catalog records or abstracts, have different distribution of terms from free-text. A modified expression for the term frequency is: tf ij = K + (1 - K)*f ij / m i when f ij > 0 K is a parameter between 0 and 1 that can be tuned for a specific collection. i

28. 28 Structured Text Query To weigh terms in the query, Salton and Buckley recommend K equal to 0.5. However, in practice it is rare for a term to be repeated in a query. Therefore the standard form of tf can be used, i.e., with K = 0 and m = 1. i

29. 29 Discussion of Similarity The choice of similarity measure is widely used and works well on a wide range of documents, but has no theoretical basis. 1.There are several possible measures other that angle between vectors 2.There is a choice of possible definitions of tf and idf 3.With fielded searching, there are various ways to adjust the weight given to each field. The definitions on Slide 25 can be considered the standard.