1. Ch 4: Information Retrieval and Text Mining
Hakam Alomari

2. 4.1: Is Information Retrieval a Form of Text Mining?
What is the principal computer specialty for processing documents and text??
Information Retrieval (IR)
The task of IR is to retrieve relevant documents in response to a query.
The fundamental technique of IR is measuring similarity
A query is examined and transformed into a vector of values to be compared with stored documents

3. Cont. 4.1
In the predication problem similar documents are retrieved, then measure their properties, i.e. count the # of class labels to see which label should be assigned to a new document
The objectives of the prediction can be posed in the form of an IR model where documents are retrieved that are relevant to a query, the query will be a new document

4. Cont. 4.1 Specify Query Search Document Collection
Return Subset of Relevant Documents
Figure 4.1. Key Steps in Information Retrieval
Examine Document Collection
Learn Classification Criteria
Apply Criteria to New Documents
Figure 4.2. Key Steps in Predictive Text Mining

5. Match Document Collection Get Subset of Relevant Documents
Specify Query Vector
Match Document Collection
Get Subset of Relevant Documents
Examine Document Properties
Figure 4.2. Key steps in IR
Figure 4.3. Predicting from Retrieved Documents
simple criteria such as document’s labels

6. 4.2 Key Word Search
The technical goal for prediction is to classify new, unseen documents
The Prediction and IR are unified by the computation of similarity of documents
IR based on traditional keyword search through a search engine
So we should recognize that using a search engine is a special instance of prediction concept

7. We enter a key words to a search engine and expect relevant documents to be returned
These key words are words in a dictionary created from the document collection and can be viewed as a small document
So, we want to measuring how similar the new document (query) is to the documents in the collection

8. So, the notion of similarity is reduced to finding documents with the same keywords as posed to the search engine
But, the objective of the search engine is to rank the documents, not to assign a label
So we need additional techniques to break the expected ties (all retrieved documents match the search criteria)

9. 4.3 Nearest-Neighbor Methods
A method that compares vectors and measures similarity
In Prediction: the NNMs will collect the K most similar documents and then look at their labels
In IR: the NNMs will determine whether a satisfactory response to the search query has been found

10. 4.4 Measuring Similarity
These measures used to examine how documents are similar and the output is a numerical measure of similarity
Three increasingly complex measures:
Shared Word Count
Word Count and Bonus
Cosine Similarity

11. 4.4.1 Shared Word Count Counts the shared words between documents
The words:
In IR we have a global dictionary where all potential words will be included, with the exception of stopwords.
In Prediction its better to preselect the dictionary relative to the label

12. Computing similarity by Shared words
Look at all words in the new document
For each document in the collection count how many of these words appear
No weighting are used, just a simple count
The dictionary has true key words (weakly words removed)
The results of this measure are clearly intuitive
No one will question why a document was retrieved

13. Computing similarity by Shared words
Each document represented as a vector of key words (zeros and ones)
The similarity of 2 documents is the product of the 2 vectors
If 2 documents have the same key word then this word is counted (1*1)
The performance of this measure depends mainly on the dictionary used

14. Computing similarity by Shared words
Shared words is an exact search
either retrieving or not retrieving a document.
No weighting can be done on terms
in query, A and B, you can’t specify A is more important than B
Every retrieved document are treated equally

15. 4.4.2 Word Count and Bonus 1/4 TF – term frequency
number of times a term occurs in a document
DF –Document frequency
Number of documents that contain the term.
IDF – inversed document frequency
=log (N/df)
N: the total number of documents
Vector is a numerical representation for a point in a multi-dimensional space.
(x1, x2, … … xn)
Dimensions of the space need to be defined
A measure of the space needs to be defined.

16. If word (j) occurs in both documents
4.4.2 Word Count and Bonus 2/4
Each indexing term is a dimension
Each document is a vector
Di = (ti1, ti2, ti3, ti4, ... tik)
Document similarity is defined as
K = number of words
If word (j) occurs in both documents
otherwise

17. 4.4.2 Word Count and Bonus 3/4
The bonus 1/df(j) is a variant of idf. Thus, if the word occurs in many documents, the bonus is small.
This measure better than the Shared Word count, because its discriminate among the weak and strong predictive words.

18. 4.4.2 Word Count and Bonus 4/4 Labeled Spreadsheet Similarity Scores
A document Space is defined by five terms: hardware, software, user, information, index.
The query is “ hardware, user, information. 1
2.83
1.33
1.5
2.67
New
Document
Vector
Measure Similarity With Bonus 1
Figure 4.4. Computing Similarity Scores with Bonus

19. 4.4.3 Cosine Similarity The Vector Space
A document is represented as a vector:
(W1, W2, … … , Wn)
Binary:
Wi= 1 if the corresponding term is in the document
Wi= 0 if the term is not in the document
TF: (Term Frequency)
Wi= tfi where tfi is the number of times the term occurred in the document
TF*IDF: (Inverse Document Frequency)
Wi =tfi*idfi=tfi*(1+log(N/dfi)) where dfi is the number of documents contains the term i, and N the total number of documents in the collection.

20. 4.4.3 Cosine Similarity The Vector Space
vec(D) = (w1, w2, ..., wt)
Sim(d1,d2) = cos()
= [vec(d1)  vec(d2)] / |d1| * |d2| = [ wd1(j) * wd2(j)] / |d1| * |d2|
W(j) > 0 whenever j di
So, 0 <= sim(d1,d2) <=1
A document is retrieved
even if it matches the
query terms only partially

21. 4.4.3 Cosine Similarity How to compute the weight wj?
A good weight must take into account two effects:
quantification of intra-document contents (similarity)
tf factor, the term frequency within a document
quantification of inter-documents separation (dissi-milarity)
idf factor, the inverse document frequency
wj = tf(j) * idf(j)

22. 4.4.3 Cosine Similarity
TF in the given document shows how important the term is in this document (makes the frequent words for the document more important)
IDF makes rare words across all documents more important.
A high weight in a tf-idf ranking scheme is therefore reached by a high term frequency in the given document and a low term frequency in all other documents.
Term weights in a document affects the position of the document vectors
di = (wi,1 , wi,2 ….wi,t)

23. 4.4.3 Cosine Similarity TF-IDF definitions:
fik: number occurrences of term ti in document Dk
tfik: fik / max(fik) normalized term frequency
dfk: number of documents which contain tk
idfk: log(N / dfk) where N is the total number of documents
wik: tfik idfk term weight
Intuition: rare words get more weight, common words less weight

24. Example TF-IDF
Given a document containing terms with given frequencies:
Kent = 3; Ohio = 2; University = 1
and assume a collection of 10,000 documents and document frequencies of these terms are:
Kent = 50; Ohio = 1300; University = 250.
THEN
Kent: tf = 3/3; idf = log(10000/50) = 5.3; tf-idf = 5.3
Ohio: tf = 2/3; idf = log(10000/1300) = 2.0; tf-idf = 1.3
University: tf = 1/3; idf = log(10000/250) = 3.7; tf-idf = 1.2

25. 4.4.3 Cosine Similarity Cosine W(j) = tf(j) * idf(j)
Idf(j) = log(N / df(j))