1. Representation of documents and queries
Why do this?
Want to compare documents
Want to compare documents with queries
Want to retrieve and rank documents with regards to a specific query
A document representation permits this in a consistent way (type of conceptualization)

2. Boolean queries
Document is relevant to a query of the query is in the document.
Document is either relevant or not relevant to the query.
What about relevance ranking – partial relevance. Vector model deals with this.

3. Matching - similarity
Define methods of similarity and matching for documents and queries
Use similarity and matching for ranking

4. Measures of similarity
Retrieve the most similar documents to a query
Equate similarity to relevance
Most similar are the most relevant
This measure is one of “lexical similarity”
The matching of text or words

5. Document space
Documents are organized in some manner - exist as points in a document space
Documents treated as text, etc.
Match query with document
Query similar to document space
Query not similar to document space and becomes a characteristic function on the document space
Documents most similar are the ones we retrieve
Reduce this a computable measure of similarity

6. Query similar to document space
Query is a point in document space
Documents “near” to the query are the ones we want.
Near:
Distance
Lying in similar direction as other documents
Others

7. Document Clustering
Term 1
Term 2

8. Documents in 3D Space Assumption: Documents that are “close together”
in space are similar in meaning.

9. Representation of Documents
Consider now only text documents
Words are tokens (primitives)
Why not letters?
Stop words?
How do we represent words?
Even for video, audio, etc documents, we often use words as part of the representation

10. Documents as Vectors Documents are represented as “bags of words”
Example?
Represented as vectors when used computationally
A vector is like an array of floating point values
Has direction and magnitude
Each vector holds a place for every term in the collection
Therefore, most vectors are sparse

11. Vector Space Model
Documents and queries are represented as vectors in term space
Terms are usually stems
Documents represented by binary vectors of terms
Queries represented the same as documents
Query and Document weights are based on length and direction of their vector
A vector distance measure between the query and documents is used to rank retrieved documents

12. The Vector-Space Model
Assume t distinct terms remain after preprocessing; call them index terms or the vocabulary.
These “orthogonal” terms form a vector space.
Dimension = t = |vocabulary|
Each term i in a document or query j is given a real-valued weight, wij.
Both documents and queries are expressed as t-dimensional vectors:
dj = (w1j, w2j, …, wtj)

13. The Vector-Space Model
3 terms, t1, t2, t3 for all documents
Vectors can be written differently
d1 = (weight of t1, weight of t2, weight of t3)
d1 = (w1,w2,w3)
d1 = w1,w2,w3 or
d1 = w1 t1 + w2 t2 + w3 t3

14. Example - documents and queries
D1: to be or not to be
D2: to be here or to be there
D3: not to be
D4: to be forever here
Q1: here
Q2: to be

15. Definitions Documents vs terms Treat documents and queries as the same
4 docs and 2 queries => 6 rows
Vocabulary in alphabetical order – dimension 7
be, forever, here, not, or, there, to => 7 columns
6 X 7 doc-term matrix
4 X 4 doc-doc matrix (exclude queries)
7 X 7 term-term matrix (exclude queries)

16. Document Collection
A collection of n documents can be represented in the vector space model by a term-document matrix.
An entry in the matrix corresponds to the “weight” of a term in the document; zero means the term has no significance in the document or it simply doesn’t exist in the document.
T1 T2 … Tt
D1 w11 w21 … wt1
D2 w12 w22 … wt2
: : : :
Dn w1n w2n … wtn
Queries are treated just like documents!

17. Assigning Weights to Terms
wij is the weight of term j in document i
Binary Weights
Raw term frequency
tf x idf
Deals with Zipf distribution
Want to weight terms highly if they are
frequent in relevant documents … BUT
infrequent in the collection as a whole

18. doc-terms matrix Binary wtsbe
forever
here
not or
there to
Doc 1 1
Doc 2
Doc 3
Doc 4
Q 1
Q 2

19. doc-doc matrix wts terms that overlap3 2
Doc 3
Doc 4

20. term-term matrix wts be
forever
here
not or
there to - 1 2 4

21. doc-terms tf wts be forever here not or there to Doc 1 2 1 Doc 2 Doc 31
Doc 2
Doc 3
Doc 4
Q 1
Q 2

22. doc-doc matrix wts term freq5 3 2
Doc 3
Doc 4

23. Term Weights: Term Frequency
More frequent terms in a document are more important, i.e. more indicative of the topic.
fij = frequency of term i in document j
May want to normalize term frequency (tf) across the entire corpus:
tfij = fij / max{fij}

24. Assigning Weights tf x idf measure:
term frequency (tf)
inverse document frequency (idf) -- a way to deal with the problems of the Zipf distribution
Goal: assign a tf x idf weight to each term in each document
A term occurring frequently in the document but rarely in the rest of the collection is given high weight.
Many other ways of determining term weights have been proposed.
Experimentally, tf-idf has been found to work well.

25. TF x IDF (term frequency-inverse document frequency)
wij = tfij [log2 (N/nj) + 1]
wij = weight of Term Tj in Document Di
tfij = frequency of Term Tj in Document Di
N = number of Documents in collection
nj = number of Documents where term Tj occurs at least once
Red text is the Inverse Document Frequency measure idfj

26. IDF (inverse document frequency)
idfj = log2 (N/nj) + 1
log2 (N/nj) + 1 = log N - log nj + 1
Recall n is 1 or greater
Moderates effect of document size

27. Inverse Document Frequency (idf)
idf provides high values for rare words and low values for common words
Double the number
of documents, what
happens?
For a collection
of documents

28. Inverse Document Frequency
idfj modifies only the columns not the rows!
log2 (N/nj) + 1 = log N - log nj + 1
Consider only the documents, not the queries!
N = 4

29. tf-idf wt calculation N = 4 be forever here not or there to Doc 1 2 11
Doc 2
Doc 3
Doc 4 n1 n2 n3 n4 n5 n6 n7 4
N = 4

30. tf-idf wt calculation N = 4 be forever here not or there to Doc 1 2 11
Doc 2
Doc 3
Doc 4 n1 n2 n3 n4 n5 n6 n7 ni 4
N/n

31. tf-idf wt calculation N = 4 be forever here not or there to Doc 1 2 11
Doc 2
Doc 3
Doc 4 n1 n2 n3 n4 n5 n6 n7 ni 4
N/n
idf 3

32. tf-idf wt calculation be forever here not or there to Doc 1 2 Doc 2 3
Doc 2 3
Doc 3 1
Doc 4

33. (Doc + queries) terms tf-idf wtsbe
forever
here
not or
there to
Doc 1 2
Doc 2 3
Doc 3 1
Doc 4
Q 1
Q 2

34. Document Similarity With a query what do we want to retrieve?
Relevant documents
Similar documents
Query should be similar to the document?
Innate concept – want a document without your query terms?

35. Similarity Measures Queries are treated like documents
Documents are ranked by some measure of closeness to the query
Closeness is determined by a Similarity Measure s
Ranking is usually s(1) > s(2) > s(3)

36. Document Similarity Types of similarity Text Content Authors
Date of creation
Images
Etc.

37. Similarity Measure - Inner Product
Similarity between vectors for the document di and query q can be computed as the vector inner product:
s = sim(dj,q) = dj•q = wij · wiq
where wij is the weight of term i in document j and wiq is the weight of term i in the query
For binary vectors, the inner product is the number of matched query terms in the document (size of intersection).
For weighted term vectors, it is the sum of the products of the weights of the matched terms.

38. Inner Product binary doc1 doc2 doc3 doc4 Q1 1 Q2 2
For binary, tf and tfidf models
binary
doc1
doc2
doc3
doc4 Q1 1 Q2 2

39. Binary wts be forever here not or there to Doc 1 1 Doc 2 Doc 3 Doc 4
Doc 2
Doc 3
Doc 4
Q 1
Q 2

40. Inner Product tf doc1 doc2 doc3 doc4 Q1 1 Q2 4 2
For binary, tf and tfidf models tf
doc1
doc2
doc3
doc4 Q1 1 Q2 4 2

41. doc-terms tf wts be forever here not or there to Doc 1 2 1 Doc 2 Doc 31
Doc 2
Doc 3
Doc 4
Q 1
Q 2

42. Inner Product tfidf doc1 doc2 doc3 doc4 Q1 4 Q2 2
For binary, tf and tfidf models
tfidf
doc1
doc2
doc3
doc4 Q1 4 Q2 2

43. (Doc + queries) terms for tf-idf wtsbe
forever
here
not or
there to
Doc 1 2
Doc 2 3
Doc 3 1
Doc 4
Q 1
Q 2

44. Inner Product binary doc1 doc2 doc3 doc4 Q1 1 Q2 2 tf doc1 doc2 doc31 Q2 2 tf
doc1
doc2
doc3
doc4 Q1 1 Q2 4 2
D1: to be or not to be
D2: to be here or to be there
D3: not to be
D4: to be forever here
Q1: here
Q2: to be
tfidf
doc1
doc2
doc3
doc4 Q1 4 Q2 2

45. Properties of Inner Product
The inner product is unbounded.
Favors long documents with a large number of unique terms.
Measures how many terms matched but not how many terms are not matched.

46. Cosine Similarity Measure2 t3 t1 t2 D1 D2 Q 1
Cosine similarity measures the cosine of the angle between two vectors.
Inner product normalized by the vector lengths.
CosSim(dj, q) =

47. Binary wts normalized be forever here not or there to Doc 1 .5 Doc 2
Doc 2
.45
Doc 3
.58
Doc 4
Q 1 1
Q 2
.71

48. Cosine Measure binary doc1 doc2 doc3 doc4 Q1 .45 .5 Q2 .71 .64 .82
For binary, tf and tfidf models
binary
doc1
doc2
doc3
doc4 Q1
.45 .5 Q2
.71
.64
.82

49. docXterms tf wts normalizedbe
forever
here
not or
there to
Doc 1
.64
.32
Doc 2
.60
.30
Doc 3
.58
Doc 4 .5
Q 1 1
Q 2
.71

50. Cosine Measure tf doc1 doc2 doc3 doc4 Q1 .30 .5 Q2 .90 .85 .82 .71
For binary, tf and tfidf models tf
doc1
doc2
doc3
doc4 Q1
.30 .5 Q2
.90
.85
.82
.71

51. (Doc + queries) terms tf-idf wts normalizedbe
forever
here
not or
there to
Doc 1 .5
Doc 2 .4 .6
Doc 3
.41
.82
Doc 4
.26
.77
.58
Q 1
.71
Q 2

52. Cosine Measure tfidf doc1 doc2 doc3 doc4 Q1 .28 .41 Q2 .71 .57 .58 .37
For binary, tf and tfidf models
tfidf
doc1
doc2
doc3
doc4 Q1
.28
.41 Q2
.71
.57
.58
.37

53. Cosine Measure bnry d1 d2 d3 d4 Q1 .45 .5 Q2 .71 .64 .82 tf d1 d2 d3
.45 .5 Q2
.71
.64
.82 tf d1 d2 d3 d4 Q1
.30 .5 Q2
.90
.85
.82
.71
D1: to be or not to be
D2: to be here or to be there
D3: not to be
D4: to be forever here
Q1: here
Q2: to be
tfidf d1 d2 d3 d4 Q1
.28
.41 Q2
.71
.57
.58
.37

54. Similarity Measures Simple matching (coordination level match)
Dice’s Coefficient
Jaccard’s Coefficient
Cosine Coefficient
Overlap Coefficient

55. Properties of similarity or matching metrics
is the similarity measure
Symmetric
(Di,Dk) = (Dk,Di)
s is close to 1 if similar
s is close to 0 if different
Others?

56. Similarity Measures
A similarity measure is a function which computes the degree of similarity between a pair of vectors or documents
since queries and documents are both vectors, a similarity measure can represent the similarity between two documents, two queries, or one document and one query
There are a large number of similarity measures proposed in the literature, because the best similarity measure doesn't exist (yet!)
With similarity measure between query and documents
it is possible to rank the retrieved documents in the order of presumed importance
it is possible to enforce certain threshold so that the size of the retrieved set can be controlled
the results can be used to reformulate the original query in relevance feedback (e.g., combining a document vector with the query vector)