1. The Vector Space Model LBSC 796/CMSC828o Session 3, February 9, 2004 Douglas W. Oard

2. Agenda Thinking about search –Design strategies –Decomposing the search component Boolean “free text” retrieval –The “bag of terms” representation –Proximity operators Ranked retrieval –Vector space model –Passage retrieval

3. Supporting the Search Process Source Selection Search Query Selection Ranked List Examination Document Delivery Document Query Formulation IR System Indexing Index Acquisition Collection     

4. Design Strategies Foster human-machine synergy –Exploit complementary strengths –Accommodate shared weaknesses Divide-and-conquer –Divide task into stages with well-defined interfaces –Continue dividing until problems are easily solved Co-design related components –Iterative process of joint optimization

5. Human-Machine Synergy Machines are good at: –Doing simple things accurately and quickly –Scaling to larger collections in sublinear time People are better at: –Accurately recognizing what they are looking for –Evaluating intangibles such as “quality” Both are pretty bad at: –Mapping consistently between words and concepts

6. Divide and Conquer Strategy: use encapsulation to limit complexity Approach: –Define interfaces (input and output) for each component Query interface: input terms, output representation –Define the functions performed by each component Remove common words, weight rare terms higher, … –Repeat the process within components as needed Result: a hierarchical decomposition

7. Search Goal Choose the same documents a human would –Without human intervention (less work) –Faster than a human could (less time) –As accurately as possible (less accuracy) Humans start with an information need –Machines start with a query Humans match documents to information needs –Machines match document & query representations

8. Search Component Model Comparison Function Representation Function Query Formulation Human Judgment Representation Function Retrieval Status Value Utility Query Information NeedDocument Query RepresentationDocument Representation Query Processing Document Processing

9. Relevance Relevance relates a topic and a document –Duplicates are equally relevant, by definition –Constant over time and across users Pertinence relates a task and a document –Accounts for quality, complexity, language, … Utility relates a user and a document –Accounts for prior knowledge We seek utility, but relevance is what we get!

10. “Bag of Terms” Representation Bag = a “set” that can contain duplicates  “The quick brown fox jumped over the lazy dog’s back”  {back, brown, dog, fox, jump, lazy, over, quick, the, the} Vector = values recorded in any consistent order  {back, brown, dog, fox, jump, lazy, over, quick, the, the}  [1 1 1 1 1 1 1 1 2]

11. Bag of Terms Example The quick brown fox jumped over the lazy dog’s back. Document 1 Document 2 Now is the time for all good men to come to the aid of their party. the quick brown fox over lazy dog back now is time for all good men to come jump aid of their party 0 0 1 1 0 1 1 0 1 1 0 0 1 0 1 0 0 1 1 0 0 1 0 0 1 0 0 1 1 0 1 0 1 1 Term Document 1Document 2 Stopword List

12. Boolean “Free Text” Retrieval Limit the bag of words to “absent” and “present” –“Boolean” values, represented as 0 and 1 Represent terms as a “bag of documents” –Same representation, but rows rather than columns Combine the rows using “Boolean operators” –AND, OR, NOT Result set: every document with a 1 remaining

13. Boolean Operators 01 11 01 0 1 A OR B A AND BA NOT B A B 00 01 01 0 1 A B 00 10 01 0 1 A B 10 01 B NOT B (= A AND NOT B)

14. Boolean Free Text Example quick brown fox over lazy dog back now time all good men come jump aid their party 0 0 1 1 0 0 0 0 0 1 0 0 1 0 1 1 0 0 1 0 0 1 0 0 1 0 0 1 1 0 0 0 0 1 Term Doc 1 Doc 2 0 0 1 1 0 1 1 0 1 1 0 0 1 0 1 0 0 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 0 1 Doc 3Doc 4 0 0 0 1 0 1 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 Doc 5Doc 6 0 0 1 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 0 0 1 0 0 1 1 1 1 0 0 0 Doc 7Doc 8 dog AND fox –Doc 3, Doc 5 dog NOT fox –Empty fox NOT dog –Doc 7 dog OR fox –Doc 3, Doc 5, Doc 7 good AND party –Doc 6, Doc 8 good AND party NOT over –Doc 6

15. Why Boolean Retrieval Works Boolean operators approximate natural language –Find documents about a good party that is not over AND can discover relationships between concepts –good party OR can discover alternate terminology –excellent party NOT can discover alternate meanings –Democratic party

16. The Perfect Query Paradox Every information need has a perfect doc set –If not, there would be no sense doing retrieval Almost every document set has a perfect query –AND every word to get a query for document 1 –Repeat for each document in the set –OR every document query to get the set query But users find Boolean query formulation hard –They get too much, too little, useless stuff, …

17. Why Boolean Retrieval Fails Natural language is way more complex –She saw the man on the hill with a telescope AND “discovers” nonexistent relationships –Terms in different paragraphs, chapters, … Guessing terminology for OR is hard –good, nice, excellent, outstanding, awesome, … Guessing terms to exclude is even harder! –Democratic party, party to a lawsuit, …

18. Proximity Operators More precise versions of AND –“NEAR n” allows at most n-1 intervening terms –“WITH” requires terms to be adjacent and in order Easy to implement, but less efficient –Store a list of positions for each word in each doc Stopwords become very important! –Perform normal Boolean computations Treat WITH and NEAR like AND with an extra constraint

19. Proximity Operator Example time AND come –Doc 2 time (NEAR 2) come –Empty quick (NEAR 2) fox –Doc 1 quick WITH fox –Empty quick brown fox over lazy dog back now time all good men come jump aid their party 01 (9) Term 1 (13) 1 (6) 1 (7) 1 (8) 1 (16) 1 (1) 1 (2) 1 (15) 1 (4) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 (5) 1 (9) 1 (3) 1 (4) 1 (8) 1 (6) 1 (10) Doc 1Doc 2

20. Strengths and Weaknesses Strong points –Accurate, if you know the right strategies –Efficient for the computer Weaknesses –Often results in too many documents, or none –Users must learn Boolean logic –Sometimes finds relationships that don’t exist –Words can have many meanings –Choosing the right words is sometimes hard

21. Ranked Retrieval Paradigm Exact match retrieval often gives useless sets –No documents at all, or way too many documents Query reformulation is one “solution” –Manually add or delete query terms “Best-first” ranking can be superior –Select every document within reason –Put them in order, with the “best” ones first –Display them one screen at a time

22. Advantages of Ranked Retrieval Closer to the way people think –Some documents are better than others Enriches browsing behavior –Decide how far down the list to go as you read it Allows more flexible queries –Long and short queries can produce useful results

23. Ranked Retrieval Challenges “Best first” is easy to say but hard to do! –The best we can hope for is to approximate it Will the user understand the process? –It is hard to use a tool that you don’t understand Efficiency becomes a concern –Only a problem for long queries, though

24. Partial-Match Ranking Form several result sets from one long query –Query for the first set is the AND of all the terms –Then all but the 1st term, all but the 2nd, … –Then all but the first two terms, … –And so on until each single term query is tried Remove duplicates from subsequent sets Display the sets in the order they were made –Document rank within a set is arbitrary

25. Partial Match Example information AND retrieval Readings in Information Retrieval Information Storage and Retrieval Speech-Based Information Retrieval for Digital Libraries Word Sense Disambiguation and Information Retrieval information NOT retrieval The State of the Art in Information Filtering Inference Networks for Document Retrieval Content-Based Image Retrieval Systems Video Parsing, Retrieval and Browsing An Approach to Conceptual Text Retrieval Using the EuroWordNet … Cross-Language Retrieval: English/Russian/French retrieval NOT information

26. Similarity-Based Queries Treat the query as if it were a document –Create a query bag-of-words Find the similarity of each document –Using the coordination measure, for example Rank order the documents by similarity –Most similar to the query first Surprisingly, this works pretty well! –Especially for very short queries

27. Document Similarity How similar are two documents? –In particular, how similar is their bag of words? 1 1 1 1: Nuclear fallout contaminated Montana. 2: Information retrieval is interesting. 3: Information retrieval is complicated. 1 1 1 1 1 1 nuclear fallout siberia contaminated interesting complicated information retrieval 1 123

28. The Coordination Measure Count the number of terms in common –Based on Boolean bag-of-words Documents 2 and 3 share two common terms –But documents 1 and 2 share no terms at all Useful for “more like this” queries –“more like doc 2” would rank doc 3 ahead of doc 1 Where have you seen this before?

29. Coordination Measure Example 1 1 1 1 1 1 1 1 1 nuclear fallout siberia contaminated interesting complicated information retrieval 1 123 Query: complicated retrieval Result: 3, 2 Query: information retrieval Result: 2, 3 Query: interesting nuclear fallout Result: 1, 2

30. Counting Terms Terms tell us about documents –If “rabbit” appears a lot, it may be about rabbits Documents tell us about terms –“the” is in every document -- not discriminating Documents are most likely described well by rare terms that occur in them frequently –Higher “term frequency” is stronger evidence –Low “collection frequency” makes it stronger still

31. The Document Length Effect Humans look for documents with useful parts –But probabilities are computed for the whole Document lengths vary in many collections –So probability calculations could be inconsistent Two strategies –Adjust probability estimates for document length –Divide the documents into equal “passages”

32. Incorporating Term Frequency High term frequency is evidence of meaning –And high IDF is evidence of term importance Recompute the bag-of-words –Compute TF * IDF for every element

33. Weighted Matching Schemes Unweighted queries –Add up the weights for every matching term User specified query term weights –For each term, multiply the query and doc weights –Then add up those values Automatically computed query term weights –Most queries lack useful TF, but IDF may be useful –Used just like user-specified query term weights

34. TF*IDF Example 4 5 6 3 1 3 1 6 5 3 4 3 7 1 nuclear fallout siberia contaminated interesting complicated information retrieval 2 123 2 3 2 4 4 0.50 0.63 0.90 0.13 0.60 0.75 1.51 0.38 0.50 2.11 0.13 1.20 123 0.60 0.38 0.50 4 0.301 0.125 0.602 0.301 0.000 0.602 Unweighted query: contaminated retrieval Result: 2, 3, 1, 4 Weighted query: contaminated(3) retrieval(1) Result: 1, 3, 2, 4 IDF-weighted query: contaminated retrieval Result: 2, 3, 1, 4

35. Document Length Normalization Long documents have an unfair advantage –They use a lot of terms So they get more matches than short documents –And they use the same words repeatedly So they have much higher term frequencies Normalization seeks to remove these effects –Related somehow to maximum term frequency –But also sensitive to the of number of terms

36. “Cosine” Normalization Compute the length of each document vector –Multiply each weight by itself –Add all the resulting values –Take the square root of that sum Divide each weight by that length

37. Cosine Normalization Example 0.29 0.37 0.53 0.13 0.62 0.77 0.57 0.14 0.19 0.79 0.05 0.71 123 0.69 0.44 0.57 4 4 5 6 3 1 3 1 6 5 3 4 3 7 1 nuclear fallout siberia contaminated interesting complicated information retrieval 2 123 2 3 2 4 4 0.50 0.63 0.90 0.13 0.60 0.75 1.51 0.38 0.50 2.11 0.13 1.20 123 0.60 0.38 0.50 4 0.301 0.125 0.602 0.301 0.000 0.602 1.700.972.670.87 Length Unweighted query: contaminated retrieval, Result: 2, 4, 1, 3 (compare to 2, 3, 1, 4)

38. Why Call It “Cosine”?  d2 d1

39. Interpreting the Cosine Measure Think of a document as a vector from zero Similarity is the angle between two vectors –Small angle = very similar –Large angle = little similarity Passes some key sanity checks –Depends on pattern of word use but not on length –Every document is most similar to itself

40. “Okapi” Term Weights TF componentIDF component

41. Passage Retrieval Another approach to long-document problem –Break it up into coherent units Recognizing topic boundaries is hard –But overlapping 300 word passages work fine Document rank is best passage rank –And passage information can help guide browsing

42. Summary Goal: find documents most similar to the query Compute normalized document term weights –Some combination of TF, DF, and Length Optionally, get query term weights from the user –Estimate of term importance Compute inner product of query and doc vectors –Multiply corresponding elements and then add

43. Before You Go! On a sheet of paper, please briefly answer the following question (no names): What was the muddiest point in today’s lecture?