Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 15-505 Internet Search Technologies Lecture 5: Information Retrieval Kamal Nigam Slides adapted from Chris Manning, Prabhakar Raghavan, and Hinrich Schütze.

Similar presentations

Presentation on theme: "1 15-505 Internet Search Technologies Lecture 5: Information Retrieval Kamal Nigam Slides adapted from Chris Manning, Prabhakar Raghavan, and Hinrich Schütze."— Presentation transcript:

1 1 15-505 Internet Search Technologies Lecture 5: Information Retrieval Kamal Nigam Slides adapted from Chris Manning, Prabhakar Raghavan, and Hinrich Schütze at

2 2 Information Retrieval in 1650 Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? One could grep all of Shakespeare’s plays for Brutus and Caesar, then strip out lines containing Calpurnia? Slow (for large corpora) NOT Calpurnia is non-trivial Other operations (e.g., find the word Romans near countrymen) not feasible Ranked retrieval (best documents to return)

3 3 Term-document incidence 1 if play contains word, 0 otherwise Brutus AND Caesar but NOT Calpurnia

4 4 Incidence vectors So we have a 0/1 vector for each term. To answer query: take the rows for Brutus, Caesar and Calpurnia (complemented)  bitwise AND. 110100 AND 110111 AND 101111 = 100100.

5 5 Answers to query Antony and Cleopatra, Act III, Scene ii Agrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain. Hamlet, Act III, Scene ii Lord Polonius: I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me.

6 6 Search index construction Tokenizer Token stream. Friends RomansCountrymen Linguistic modules Modified tokens. friend romancountryman Indexer Inverted index. friend roman countryman 24 2 13 16 1 Documents to be indexed. Friends, Romans, countrymen.

7 7 Tokenization Tokenizer Token stream. Friends RomansCountrymen Linguistic modules Modified tokens. friend romancountryman Indexer Inverted index. friend roman countryman 24 2 13 16 1 Documents to be indexed. Friends, Romans, countrymen.

8 8 Tokenization Input: “Friends, Romans and Countrymen” Output: Tokens Friends Romans and Countrymen Each token is candidate for an index entry, after further processing But what are valid tokens to emit?

9 9 Tokenization: tricky cases Finland’s  Finland? Finlands? Finland’s? Hewlett-Packard  Hewlett and Packard as one or two tokens? the hold-him-back-and-drag-him-away-maneuver ? Numbers: unique token for every number? L'ensemble  one token or two? L ? L’ ? Le ? German noun compounds are not segmented Lebensversicherungsgesellschaftsangestellter ‘life insurance company employee’ Chinese and Japanese have no spaces between words: 莎拉波娃现在居住在美国东南部的佛罗里达。

10 10 Normalization Tokenizer Token stream. Friends RomansCountrymen Linguistic modules Modified tokens. friend romancountryman Indexer Inverted index. friend roman countryman 24 2 13 16 1 Documents to be indexed. Friends, Romans, countrymen.

11 11 Normalization Need to “normalize” terms in indexed text as well as query terms into the same form We want to match U.S.A. and USA We most commonly implicitly define equivalence classes of terms e.g., by deleting periods in a term Alternative is to do asymmetric expansion: Enter: windowSearch: window, windows Enter: windowsSearch: Windows, windows Enter: WindowsSearch: Windows Potentially more powerful, but less efficient

12 12 Case folding Reduce all letters to lower case exception: upper case (in mid-sentence?) e.g., General Motors Fed vs. fed SAIL vs. sail Often best to lower case everything, since users will use lowercase regardless of ‘correct’ capitalization… Makes some queries hard: “The Gap” – clothing store

13 13 Stop words With a stop list, you exclude from dictionary entirely the commonest words. Intuition: They have little semantic content: the, a, and, to, be They take a lot of space: ~30% of postings for top 30 But the trend is away from doing this: Good compression lets the space for stopwords be very small Good query optimization techniques mean you pay little at query time for including stop words. You need them for: Phrase queries: “King of Denmark” Various song titles, etc.: “Let it be”, “To be or not to be” “Relational” queries: “flights to London”

14 14 Stemming Reduce terms to their “roots” before indexing “Stemming” suggest crude affix chopping language dependent e.g., automate(s), automatic, automation all reduced to automat. for example compressed and compression are both accepted as equivalent to compress. for exampl compress and compress ar both accept as equival to compress

15 15 Porter’s algorithm Commonest algorithm for stemming English Results suggest at least as good as other stemming options Conventions + 5 phases of reductions phases applied sequentially each phase consists of a set of commands sample convention: Of the rules in a compound command, select the one that applies to the longest suffix. ies  i ational  ate tional  tion

16 16 Normalization: other languages Accents: résumé vs. resume. Most important criterion: How are your users like to write their queries for these words? Even in languages with common accents, users often may not type them German: Tuebingen vs. T ü bingen Should be equivalent

17 17 Normalizations Full morphological analysis – at most modest benefits for retrieval Do stemming and other normalizations help? Often very mixed results: really help recall for some queries but harm precision on others

18 18 Indexing Tokenizer Token stream. Friends RomansCountrymen Linguistic modules Modified tokens. friend romancountryman Indexer Inverted index. friend roman countryman 24 2 13 16 1 Documents to be indexed. Friends, Romans, countrymen.

19 19 Big(ger) corpora Consider N = 1M documents, each with about 1K terms. ~6 bytes/term including spaces/punctuation 6GB of data in the documents. Say there are m = 500K distinct terms among these. 500K x 1M matrix has half-a-trillion 0’s and 1’s. But it has no more than one billion 1’s. matrix is extremely sparse. What’s a better representation? We only record the 1 positions. Why?

20 20 Inverted index For each term T, we must store a list of all documents that contain T. Do we use an array or a list for this? Brutus Calpurnia Caesar 12358132134 248163264128 1316 What happens if the word Caesar is added to document 14?

21 21 Inverted index Linked lists generally preferred to arrays Dynamic space allocation Insertion of terms into documents easy Space overhead of pointers Brutus Calpurnia Caesar 248163264128 2358132134 1316 1 Dictionary Postings lists Sorted by docID (more later on why). Posting

22 22 Sequence of (Modified token, Document ID) pairs. I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Doc 1 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious Doc 2 Indexer steps

23 23 Sort by terms. Core indexing step. Indexer steps

24 24 Multiple term entries in a single document are merged. Frequency information is added. Why frequency? Will discuss later. Indexer steps

25 25 The result is split into a Dictionary file and a Postings file.

26 26 Where do we pay in storage? Pointers Terms

27 27 Retrieval

28 28 Query processing We’ve just build something cool. How do we use it?

29 29 Query processing: AND Consider processing the query: Brutus AND Caesar Locate Brutus in the Dictionary; Retrieve its postings. Locate Caesar in the Dictionary; Retrieve its postings. “Merge” the two postings: 128 34 248163264123581321 Brutus Caesar

30 30 34 12824816 3264 12 3 581321 The merge Walk through the two postings simultaneously, in time linear in the total number of postings entries 128 34 248163264123581321 Brutus Caesar 2 8 If the list lengths are x and y, the merge takes O(x+y) operations. Crucial: postings sorted by docID.

31 31 Boolean queries: Exact match Boolean Queries are queries using AND, OR and NOT to join query terms Views each document as a set of words Is precise: document matches condition or not. Primary commercial retrieval tool for 3 decades. Professional searchers (e.g., lawyers) still like Boolean queries: You know exactly what you’re getting.

32 32 Boolean queries: More general merges Exercise: Adapt the merge for the queries: Brutus AND NOT Caesar Brutus OR NOT Caesar Can we still run through the merge in time O(x+y) or what can we achieve?

33 33 Merging What about an arbitrary Boolean formula? (Brutus OR Caesar) AND NOT (Antony OR Cleopatra) Can we always merge in “linear” time? Linear in what?

34 34 Query optimization What is the best order for query processing? Consider a query that is an AND of t terms. For each of the t terms, get its postings, then AND them together. Brutus Calpurnia Caesar 12358162134 248163264128 1316 Query: Brutus AND Calpurnia AND Caesar

35 35 Query optimization example Process in order of increasing freq: start with smallest set, then keep cutting further. Brutus Calpurnia Caesar 12358132134 248163264128 1316 This is why we kept freq in dictionary Execute the query as (Caesar AND Brutus) AND Calpurnia.

36 36 Phrase queries

37 37 Phrase queries Want to answer queries such as “carnegie mellon” – as a phrase Thus the sentence “I got money from the Mellon bank in Carnegie” is not a match. The concept of phrase queries has proven easily understood by users; about 10% of web queries are phrase queries No longer suffices to store only entries

38 38 Positional indexes Store, for each term, entries of the form:

39 39 Positional index example Can compress position values/offsets Nevertheless, this expands postings storage substantially Which of docs 1,2,4,5 could contain “to be or not to be”?

40 40 Processing a phrase query Extract inverted index entries for each distinct term: to, be, or, not. Merge their doc:position lists to enumerate all positions with “to be or not to be”. to: 2:1,17,74,222,551; 4:8,16,190,429,433; 7:13,23,191;... be: 1:17,19; 4:17,191,291,430,434; 5:14,19,101;... Same general method for proximity searches

41 41 Positional index size You can compress position values/offsets Still, a positional index expands postings storage substantially Standardly used because of the power and usefulness of phrase and proximity queries … whether used explicitly or implicitly in a ranking retrieval system.

42 42 Rules of thumb A positional index is 2 – 4 as large as a non- positional index Positional index size 35 – 50% of volume of original text Caveat: all of this holds for “English-like” languages

43 43 Scoring and Ranking

44 44 Scoring Thus far, our queries have all been Boolean Docs either match or not Good for expert users with precise understanding of their needs and the corpus Applications can consume 1000’s of results Not good for (the majority of) users with poor Boolean formulation of their needs Most users don’t want to wade through 1000’s of results – cf. use of web search engines

45 45 Scoring We wish to return in order the documents most likely to be useful to the searcher How can we rank order the docs in the corpus with respect to a query? Assign a score – say in [0,1] for each doc on each query Assume no spammers “adversarial IR” makes life complicated

46 46 Incidence matrices Recall: Document is binary vector X in {0,1} v Query is a binary vector Y Score: Overlap measure:

47 47 Example On the query ides of march, Shakespeare’s Julius Caesar has a score of 3 All other Shakespeare plays have a score of 2 (because they contain march) or 1 Thus in a rank order, Julius Caesar would come out tops

48 48 Overlap matching What’s wrong with the overlap measure? It doesn’t consider: Term frequency in document Term scarcity in collection (document mention frequency) of is more common than ides or march Length of documents (And queries: score not normalized)

49 49 Scoring: density-based Thus far: overlap of terms/phrases in a doc Next: if a document talks about a topic more, then it is a better match Document relevant if it has a lot of the terms This leads to the idea of term weighting.

50 50 Scoring: Term weighting

51 51 Term-document count matrices Consider the number of occurrences of a term in a document: Bag of words model Document is a vector in ℕ v : a column below

52 52 Bag of words view of a doc Thus the doc John is quicker than Mary. is indistinguishable from the doc Mary is quicker than John.

53 53 Term frequency tf Idea: Score by summing all occurrences of query words in doc Long docs are favored because they’re more likely to contain query terms Can fix this to some extent by normalizing for document length But is raw tf the right measure? Note: in IR frequency means unnormalized count

54 54 Counts vs. frequencies Consider again the ides of march query. Julius Caesar has 5 occurrences of ides No other play has ides march occurs in over a dozen All the plays contain lots of of By this scoring measure, the top-scoring play is likely to be the one with the most ofs

55 55 Weighting term frequency: tf What is the relative importance of 0 vs. 1 occurrence of a term in a doc 1 vs. 2 occurrences 2 vs. 3 occurrences … Unclear: while it seems that more is better, a lot isn’t proportionally better than a few Can just use raw tf Another option commonly used in practice: Note: in IR frequency means unnormalized count

56 56 Score computation Score for a query q = sum over terms t in q: [Note: 0 if no query terms in document] Can use wf instead of tf in the above Still doesn’t consider term scarcity in collection (ides is rarer than of)

57 57 Weighting should depend on the term overall Which of these tells you more about a doc? 10 occurrences of ides? 10 occurrences of of? Attenuate the weight of a common term But what is “common”? Consider collection frequency (cf ) The total number of occurrences of the term in the entire collection of documents

58 58 Document frequency But document frequency (df ) may be better: df = number of docs in the corpus containing the term Wordcfdf ferrari1042217 insurance104403997 Document/collection frequency weighting is only possible in known (static) collection. So how do we make use of df ?

59 59 tf x idf term weights tf x idf measure combines: term frequency (tf ) or wf, some measure of term density in a doc inverse document frequency (idf ) measure of informativeness of a term: its rarity across the whole corpus The most commonly used version is:

60 60 Summary: tf x idf (or tf.idf) Assign a tf.idf weight to each term i in each document d Increases with the number of occurrences within a doc Increases with the rarity of the term across the whole corpus What is the wt of a term that occurs in all of the docs?

61 61 Real-valued term-document matrices Function (scaling) of count of a word in a document: Bag of words model Each is a vector in ℝ v Here log-scaled tf.idf Note can be >1!

62 62 Scoring: similarity

63 63 Documents as vectors Each doc j can now be viewed as a vector of wf  idf values, one component for each term So we have a vector space terms are axes docs live in this space even with stemming, this is huge-dimensional (The corpus of documents gives us a matrix, which we could also view as a vector space in which words live – transposable data)

64 64 Why turn docs into vectors? You can also turn queries into (short) vectors With q a vector, find d vectors (docs) “ near ” it.

65 65 Intuition Postulate: Text that is “close together” in the vector space talk about similar things. t1t1 d2d2 q d3d3 d4d4 d5d5 t3t3 t2t2

66 66 First cut at similarity Idea: Distance between d 1 and d 2 is the length of the vector |d 1 – d 2 |. Euclidean distance Why is this not a great idea? We still haven ’ t dealt with the issue of length normalization Short documents would be more similar to each other by virtue of length, not topic However, we can implicitly normalize by looking at angles instead

67 67 Cosine similarity Distance between vectors d 1 and d 2 captured by the cosine of the angle x between them. Note – this is similarity, not distance t 1 d 2 d 1 t 3 t 2 θ

68 68 Cosine similarity A vector can be normalized (given a length of 1) by dividing each of its components by its length – here we use the L 2 norm This maps vectors onto the unit sphere: Then, Longer documents don ’ t get more weight

69 69 Cosine similarity Cosine of angle between two vectors The denominator involves the lengths of the vectors. Normalization

70 70 Normalized vectors For normalized vectors, the cosine is simply the dot product:

71 71 Queries in the vector space model Central idea: the query as a vector: We regard the query as short document We return the documents ranked by the closeness of their vectors to the query, also represented as a vector. Note that d q is very sparse!

72 72 What did we learn today? Length-normalized TFIDF-weighted/positional inverted index to compute cosine similarities Say it three times fast Store, for each term, entries of the form: Compute boolean matched documents with dot product scores together, and present them ranked to user

73 73 Following up on today’s lecture Free online textbook: http://www- book.html Good in-depth classes at CMU: 15-493 & 11-741 Taught by Jamie Callan and Yiming Yang

Download ppt "1 15-505 Internet Search Technologies Lecture 5: Information Retrieval Kamal Nigam Slides adapted from Chris Manning, Prabhakar Raghavan, and Hinrich Schütze."

Similar presentations

Ads by Google