1. 2011.01.31- SLIDE 1IS 240 – Spring 2011 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval Lecture 4: IR System Elements (cont)

2. 2011.01.31- SLIDE 2IS 240 – Spring 2011 Review –Elements of IR Systems Collections, Queries Text processing and Zipf distribution –Stemmers and Morphological analysis (cont…) Inverted file indexes IR Models - Introduction to the Boolean Model

3. 2011.01.31- SLIDE 3IS 240 – Spring 2011 Queries A query is some expression of a user’s information needs Can take many forms –Natural language description of need –Formal query in a query language Queries may not be accurate expressions of the information need –Differences between conversation with a person and formal query expression

4. 2011.01.31- SLIDE 4IS 240 – Spring 2011 Collections of Documents… Documents –A document is a representation of some aggregation of information, treated as a unit. Collection –A collection is some physical or logical aggregation of documents Let’s take the simplest case, and say we are dealing with a computer file of plain ASCII text, where each line represents the “UNIT” or document.

5. 2011.01.31- SLIDE 5IS 240 – Spring 2011 How to search that collection? Manually? –Cat, more Scan for strings? –Grep Extract individual words to search??? –“tokenize” (a unix pipeline) tr -sc ’A-Za-z’ ’\012’ < TEXTFILE | sort | uniq –c –See “Unix for Poets” by Ken Church Put it in a DBMS and use pattern matching there… –assuming the lines are smaller than the text size limits for the DBMS

6. 2011.01.31- SLIDE 6IS 240 – Spring 2011 What about VERY big files? Scanning becomes a problem The nature of the problem starts to change as the scale of the collection increases A variant of Parkinson’s Law that applies to databases is: –Data expands to fill the space available to store it

7. 2011.01.31- SLIDE 7 Document Processing Steps

8. 2011.01.31- SLIDE 8IS 240 – Spring 2011 Structure of an IR System Search Line Interest profiles & Queries Documents & data Rules of the game = Rules for subject indexing + Thesaurus (which consists of Lead-In Vocabulary and Indexing Language Storage Line Potentially Relevant Documents Comparison/ Matching Store1: Profiles/ Search requests Store2: Document representations Indexing (Descriptive and Subject) Formulating query in terms of descriptors Storage of profiles Storage of Documents Information Storage and Retrieval System Adapted from Soergel, p. 19

9. 2011.01.31- SLIDE 9IS 240 – Spring 2011 Query Processing In order to correctly match queries and documents they must go through the same text processing steps as the documents did when they were stored In effect, the query is treated like it was a document Exceptions (of course) include things like structured query languages that must be parsed to extract the search terms and requested operations from the query –The search terms must still go through the same text process steps as the document…

10. 2011.01.31- SLIDE 10IS 240 – Spring 2011 Steps in Query processing Parsing and analysis of the query text (same as done for the document text) –Morphological Analysis –Statistical Analysis of text

11. 2011.01.31- SLIDE 11IS 240 – Spring 2011 Plotting Word Frequency by Rank Say for a text with 100 tokens Count –How many tokens occur 1 time (50) –How many tokens occur 2 times (20) … –How many tokens occur 7 times (10) … –How many tokens occur 12 times (1) –How many tokens occur 14 times (1) So things that occur the most often share the highest rank (rank 1). Things that occur the fewest times have the lowest rank (rank n).

12. 2011.01.31- SLIDE 12IS 240 – Spring 2011 Many similar distributions… Words in a text collection Library book checkout patterns Bradford’s and Lotka’s laws. Incoming Web Page Requests (Nielsen) Outgoing Web Page Requests (Cunha & Crovella) Document Size on Web (Cunha & Crovella)

13. 2011.01.31- SLIDE 13 Zipf Distribution (linear and log scale)

14. 2011.01.31- SLIDE 14IS 240 – Spring 2011 Resolving Power (van Rijsbergen 79) The most frequent words are not the most descriptive.

15. 2011.01.31- SLIDE 15IS 240 – Spring 2011 Other Models Poisson distribution 2-Poisson Model Negative Binomial Katz K-mixture –See Church (SIGIR 1995)

16. 2011.01.31- SLIDE 16IS 240 – Spring 2011 Stemming and Morphological Analysis Goal: “normalize” similar words Morphology (“form” of words) –Inflectional Morphology E.g,. inflect verb endings and noun number Never change grammatical class –dog, dogs –tengo, tienes, tiene, tenemos, tienen –Derivational Morphology Derive one word from another, Often change grammatical class –build, building; health, healthy

17. 2011.01.31- SLIDE 17IS 240 – Spring 2011 Stemming and Morphological Analysis Goal: “normalize” similar words Morphology (“form” of words) –Inflectional Morphology E.g,. inflect verb endings and noun number Never change grammatical class –dog, dogs –tengo, tienes, tiene, tenemos, tienen –Derivational Morphology Derive one word from another, Often change grammatical class –build, building; health, healthy

18. 2011.01.31- SLIDE 18IS 240 – Spring 2011 Simple “S” stemming IF a word ends in “ies”, but not “eies” or “aies” –THEN “ies”  “y” IF a word ends in “es”, but not “aes”, “ees”, or “oes” –THEN “es”  “e” IF a word ends in “s”, but not “us” or “ss” –THEN “s”  NULL Harman, JASIS Jan. 1991

19. 2011.01.31- SLIDE 19IS 240 – Spring 2011 Stemmer Examples The SMART stemmer The Porter stemmer The IAGO! stemmer % tstem ate ate % tstem apples appl % tstem formulae formul % tstem appendices appendix % tstem implementation imple % tstem glasses glass % pstem ate at % pstem apples appl % pstem formulae formula % pstem appendices appendic % pstem implementation implement % pstem glasses glass % stem ate|2 eat|2 apples|1 apple|1 formulae|1 formula|1 appendices|1 appendix|1 implementation|1 glasses|1

20. 2011.01.31- SLIDE 20IS 240 – Spring 2011 Too AggressiveToo Timid organization/organ policy/police execute/executive arm/army european/europe cylinder/cylindrical create/creation search/searcher Errors Generated by Porter Stemmer (Krovetz 93)

21. 2011.01.31- SLIDE 21IS 240 – Spring 2011 Automated Methods Stemmers: –Very dumb rules work well (for English) –Porter Stemmer: Iteratively remove suffixes –Improvement: pass results through a lexicon Newer stemmers are configurable (Snowball) –Demo… Powerful multilingual tools exist for morphological analysis –PCKimmo, Xerox Lexical technology –Require a grammar and dictionary –Use “two-level” automata –Wordnet “morpher”

22. 2011.01.31- SLIDE 22IS 240 – Spring 2011 Wordnet Type “wn word” on a machine where wordnet is installed… Large exception dictionary: Demo aardwolves aardwolf abaci abacus abacuses abacus abbacies abbacy abhenries abhenry abilities ability abkhaz abnormalities abnormality aboideaus aboideau aboideaux aboideau aboiteaus aboiteau aboiteaux aboiteau abos abo abscissae abscissa abscissas abscissa absurdities absurdity …

23. 2011.01.31- SLIDE 23IS 240 – Spring 2011 Using NLP Strzalkowski (in Reader) TextNLPrepres Dbase search TAGGER NLP: PARSERTERMS

24. 2011.01.31- SLIDE 24IS 240 – Spring 2011 Using NLP INPUT SENTENCE The former Soviet President has been a local hero ever since a Russian tank invaded Wisconsin. TAGGED SENTENCE The/dt former/jj Soviet/jj President/nn has/vbz been/vbn a/dt local/jj hero/nn ever/rb since/in a/dt Russian/jj tank/nn invaded/vbd Wisconsin/np./per

25. 2011.01.31- SLIDE 25IS 240 – Spring 2011 Using NLP TAGGED & STEMMED SENTENCE the/dt former/jj soviet/jj president/nn have/vbz be/vbn a/dt local/jj hero/nn ever/rb since/in a/dt russian/jj tank/nn invade/vbd wisconsin/np./per

26. 2011.01.31- SLIDE 26IS 240 – Spring 2011 Using NLP PARSED SENTENCE [assert [[perf [have]][[verb[BE]] [subject [np[n PRESIDENT][t_pos THE] [adj[FORMER]][adj[SOVIET]]]] [adv EVER] [sub_ord[SINCE [[verb[INVADE]] [subject [np [n TANK][t_pos A] [adj [RUSSIAN]]]] [object [np [name [WISCONSIN]]]]]]]]]

27. 2011.01.31- SLIDE 27IS 240 – Spring 2011 Using NLP EXTRACTED TERMS & WEIGHTS President 2.623519 soviet 5.416102 President+soviet 11.556747 president+former 14.594883 Hero 7.896426 hero+local 14.314775 Invade 8.435012 tank 6.848128 Tank+invade 17.402237 tank+russian 16.030809 Russian 7.383342 wisconsin 7.785689

28. 2011.01.31- SLIDE 28IS 240 – Spring 2011 Same Sentence, different sys Enju Parser ROOTROOTROOTROOT-1ROOTbeenbeVBNVB5 beenbeVBNVB5ARG1PresidentpresidentNNPNNP3 beenbeVBNVB5ARG2heroheroNNNN8 aaDTDT6ARG1heroheroNNNN8 aaDTDT11ARG1tanktankNNNN13 locallocalJJJJ7ARG1heroheroNNNN8 ThetheDTDT0ARG1PresidentpresidentNNPNNP3 formerformerJJJJ1ARG1PresidentpresidentNNPNNP3 RussianrussianJJJJ12ARG1tanktankNNNN13 SovietsovietNNPNNP2MODPresidentpresidentNNPNNP3 invadedinvadeVBDVB14ARG1tanktankNNNN13 invadedinvadeVBDVB14ARG2WisconsinwisconsinNNPNNP15 hashaveVBZVB4ARG1PresidentpresidentNNPNNP3 hashaveVBZVB4ARG2beenbeVBNVB5 sincesinceININ10MODbeenbeVBNVB5 sincesinceININ10ARG1invadedinvadeVBDVB14 evereverRBRB9ARG1sincesinceININ10

29. 2011.01.31- SLIDE 29IS 240 – Spring 2011 Other Considerations Church (SIGIR 1995) looked at correlations between forms of words in texts

30. 2011.01.31- SLIDE 30IS 240 – Spring 2011 Assumptions in IR Statistical independence of terms Dependence approximations

31. 2011.01.31- SLIDE 31IS 240 – Spring 2011 Statistical Independence Two events x and y are statistically independent if the product of their probability of their happening individually equals their probability of happening together.

32. 2011.01.31- SLIDE 32IS 240 – Spring 2011 Statistical Independence and Dependence What are examples of things that are statistically independent? What are examples of things that are statistically dependent?

33. 2011.01.31- SLIDE 33IS 240 – Spring 2011 How likely is a red car to drive by given we’ve seen a black one? How likely is the word “ambulence” to appear, given that we’ve seen “car accident”? Color of cars driving by are independent (although more frequent colors are more likely) Words in text are not independent (although again more frequent words are more likely) Statistical Independence vs. Statistical Dependence

34. 2011.01.31- SLIDE 34IS 240 – Spring 2011 Lexical Associations Subjects write first word that comes to mind –doctor/nurse; black/white (Palermo & Jenkins 64) Text Corpora yield similar associations One measure: Mutual Information (Church and Hanks 89) If word occurrences were independent, the numerator and denominator would be equal (if measured across a large collection)

35. 2011.01.31- SLIDE 35IS 240 – Spring 2011 Interesting Associations with “Doctor” (AP Corpus, N=15 million, Church & Hanks 89) I(x,y)f(x,y)f(x)xf(y)y 11.3 10.7 9.4 9.0 8.9 8.7 12 8 30 8 6 11 25 111 1105 275 1105 621 honorary doctors examined doctors doctor 621 44 241 154 621 317 1407 doctor dentists nurses treating doctor treat bills

36. 2011.01.31- SLIDE 36IS 240 – Spring 2011 These associations were likely to happen because the non-doctor words shown here are very common and therefore likely to co-occur with any noun. Un-Interesting Associations with “Doctor” I(x,y)f(x,y)f(x)xf(y)y 0.96 0.95 0.93 6 41 12 621 284690 84716 doctor a is 73785 1105 with doctors

37. 2011.01.31- SLIDE 37IS 240 – Spring 2011 Query Processing Once the text is in a form to match to the indexes then the fun begins –What approach to use? Boolean? Extended Boolean? Ranked –Fuzzy sets? –Vector? –Probabilistic? –Language Models? –Neural nets? Most of the next few weeks will be looking at these different approaches

38. 2011.01.31- SLIDE 38IS 240 – Spring 2011 Display and formatting Have to present the the results to the user Lots of different options here, mostly governed by –How the actual document is stored –And whether the full document or just the metadata about it is presented

39. 2011.01.31- SLIDE 39IS 240 – Spring 2011 What to do with terms… Once terms have been extracted from the documents, they need to be stored in some way that lets you get back to documents that those terms came from The most common index structure to do this in IR systems is the “Inverted File”

40. 2011.01.31- SLIDE 40IS 240 – Spring 2011 Boolean Implementation: Inverted Files We will look at “Vector files” in detail later. But conceptually, an Inverted File is a vector file “inverted” so that rows become columns and columns become rows

41. 2011.01.31- SLIDE 41IS 240 – Spring 2011 How Are Inverted Files Created Documents are parsed to extract words (or stems) and these are saved with the Document ID. Now is the time for all good men to come to the aid of their country Doc 1 It was a dark and stormy night in the country manor. The time was past midnight Doc 2 Text Proc Steps

42. 2011.01.31- SLIDE 42IS 240 – Spring 2011 How Inverted Files are Created After all document have been parsed the inverted file is sorted

43. 2011.01.31- SLIDE 43IS 240 – Spring 2011 How Inverted Files are Created Multiple term entries for a single document are merged and frequency information added

44. 2011.01.31- SLIDE 44IS 240 – Spring 2011 Inverted Files The file is commonly split into a Dictionary and a Postings file

45. 2011.01.31- SLIDE 45IS 240 – Spring 2011 Inverted files Permit fast search for individual terms Search results for each term is a list of document IDs (and optionally, frequency and/or positional information) These lists can be used to solve Boolean queries: –country: d1, d2 –manor: d2 –country and manor: d2

46. 2011.01.31- SLIDE 46IS 240 – Spring 2011 Inverted Files Lots of alternative implementations –E.g.: Cheshire builds within-document frequency using a hash table during document parsing. Then Document IDs and frequency info are stored in a BerkeleyDB B- tree index keyed by the term.

47. 2011.01.31- SLIDE 47IS 240 – Spring 2011 Btree (conceptual) B | | D | | F | Aces Boilers Cars F | | P | | Z | R | | S | | Z |H | | L | | P | Devils Minors Panthers Seminoles Flyers Hawkeyes Hoosiers

48. 2011.01.31- SLIDE 48IS 240 – Spring 2011 Btree with Postings B | | D | | F | Aces Boilers Cars F | | P | | Z | R | | S | | Z |H | | L | | P | Devils Minors Panthers Seminoles Flyers Hawkeyes Hoosiers 2,4,8,12 5, 7, 200 2,4,8,12 8,120

49. 2011.01.31- SLIDE 49IS 240 – Spring 2011 Inverted files Permit fast search for individual terms Search results for each term is a list of document IDs (and optionally, frequency, part of speech and/or positional information) These lists can be used to solve Boolean queries: –country: d1, d2 –manor: d2 –country and manor: d2

50. 2011.01.31- SLIDE 50IS 240 – Spring 2011 Query Processing Once the text is in a form to match to the indexes then the fun begins –What approach to use? Boolean? Extended Boolean? Ranked –Fuzzy sets? –Vector? –Probabilistic? –Language Models? –Neural nets? Most of the next few weeks will be looking at these different approaches

51. 2011.01.31- SLIDE 51IS 240 – Spring 2011 Display and formatting Have to present the the results to the user Lots of different options here, mostly governed by –How the actual document is stored –And whether the full document or just the metadata about it is presented

52. 2011.01.31- SLIDE 52IS 240 – Spring 2011 Now we have a system… Except for the matching and ranking between the query representation and the document representation –Stored in the inverted files We will start to take a look at one model for matching today The Boolean Model

53. 2011.01.31- SLIDE 53IS 240 – Spring 2011 IR Models Set Theoretic Models –Boolean –Fuzzy –Extended Boolean Vector Models (Algebraic) Probabilistic Models (probabilistic) Others (e.g., neural networks, etc.)

54. 2011.01.31- SLIDE 54IS 240 – Spring 2011 Boolean Model for IR Based on Boolean Logic (Algebra of Sets). Fundamental principles established by George Boole in the 1850’s Deals with set membership and operations on sets Set membership in IR systems is usually based on whether (or not) a document contains a keyword (term)

55. 2011.01.31- SLIDE 55IS 240 – Spring 2011 Intersection – Boolean ‘AND’ -- -- Union – Boolean ‘OR’ -- -- Negation – Boolean ‘NOT’ -- -- –Usually means “AND NOT” in IR Exclusive OR – ‘XOR’ – seldom used, –Instead Boolean Operations on Sets

56. 2011.01.31- SLIDE 56IS 240 – Spring 2011 Boolean Logic AB

57. 2011.01.31- SLIDE 57IS 240 – Spring 2011 Query Languages A way to express the query (formal expression of the information need) Types: –Boolean –Natural Language –Stylized Natural Language –Form-Based (GUI)

58. 2011.01.31- SLIDE 58IS 240 – Spring 2011 Simple query language: Boolean Terms + Connectors –terms words normalized (stemmed) words phrases thesaurus terms –connectors AND OR NOT –parentheses (for grouping operations)

59. 2011.01.31- SLIDE 59IS 240 – Spring 2011 Boolean Queries Cat Cat OR Dog Cat AND Dog (Cat AND Dog) (Cat AND Dog) OR Collar (Cat AND Dog) OR (Collar AND Leash) (Cat OR Dog) AND (Collar OR Leash)

60. 2011.01.31- SLIDE 60IS 240 – Spring 2011 Boolean Queries (Cat OR Dog) AND (Collar OR Leash) –Each of the following combinations works:

61. 2011.01.31- SLIDE 61IS 240 – Spring 2011 Boolean Queries (Cat OR Dog) AND (Collar OR Leash) –None of the following combinations works:

62. 2011.01.31- SLIDE 62IS 240 – Spring 2011 Next Time More on the Boolean Model including extended Boolean approaches