1. 2009.02.23 - SLIDE 1IS 240 – Spring 2009 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval Lecture 9: Probabilistic Retrieval

2. 2009.02.23 - SLIDE 2IS 240 – Spring 2009 Mini-TREC Need to make groups –Today – Give me a note with group members (names and login names) Systems –SMART (not recommended…) ftp://ftp.cs.cornell.edu/pub/smart –MG (We have a special version if interested) http://www.mds.rmit.edu.au/mg/welcome.html –Cheshire II & 3 II = ftp://cheshire.berkeley.edu/pub/cheshire & http://cheshire.berkeley.edu 3 = http://cheshire3.sourceforge.org –Zprise (Older search system from NIST) http://www.itl.nist.gov/iaui/894.02/works/zp2/zp2.html –IRF (new Java-based IR framework from NIST) http://www.itl.nist.gov/iaui/894.02/projects/irf/irf.html –Lemur http://www-2.cs.cmu.edu/~lemur –Lucene (Java-based Text search engine) http://jakarta.apache.org/lucene/docs/index.html –Galago (Also Java-based) http://www.galagosearch.org –Others?? (See http://searchtools.com )http://searchtools.com

3. 2009.02.23 - SLIDE 3IS 240 – Spring 2009 Mini-TREC Proposed Schedule –February 16 – Database and previous Queries –March 2 – report on system acquisition and setup –March 9, New Queries for testing… –April 20, Results due –April 27, Results and system rankings –May 6 Group reports and discussion

4. 2009.02.23 - SLIDE 4IS 240 – Spring 2009 Review: IR Models Set Theoretic Models –Boolean –Fuzzy –Extended Boolean Vector Models (Algebraic) Probabilistic Models (probabilistic)

5. 2009.02.23 - SLIDE 5IS 240 – Spring 2009 Similarity Measures Simple matching (coordination level match) Dice’s Coefficient Jaccard’s Coefficient Cosine Coefficient Overlap Coefficient

6. 2009.02.23 - SLIDE 6IS 240 – Spring 2009 Documents in Vector Space t1t1 t2t2 t3t3 D1D1 D2D2 D 10 D3D3 D9D9 D4D4 D7D7 D8D8 D5D5 D 11 D6D6

7. 2009.02.23 - SLIDE 7IS 240 – Spring 2009 Vector Space Visualization

8. 2009.02.23 - SLIDE 8IS 240 – Spring 2009 Vector Space with Term Weights and Cosine Matching 1.0 0.8 0.6 0.4 0.2 0.80.60.40.201.0 D2D2 D1D1 Q Term B Term A D i =(d i1,w di1 ;d i2, w di2 ;…;d it, w dit ) Q =(q i1,w qi1 ;q i2, w qi2 ;…;q it, w qit ) Q = (0.4,0.8) D1=(0.8,0.3) D2=(0.2,0.7)

9. 2009.02.23 - SLIDE 9IS 240 – Spring 2009 Document/Document Matrix

10. 2009.02.23 - SLIDE 10IS 240 – Spring 2009 Hierarchical Methods 2.4 3.4.2 4.3.3.3 5.1.4.4.1 1 2 3 4 Single Link Dissimilarity Matrix Hierarchical methods: Polythetic, Usually Exclusive, Ordered Clusters are order-independent

11. 2009.02.23 - SLIDE 11IS 240 – Spring 2009 Threshold =.1 Single Link Dissimilarity Matrix 2.4 3.4.2 4.3.3.3 5.1.4.4.1 1 2 3 4 2 0 3 0 0 4 0 0 0 5 1 0 0 1 1 2 3 4 2 1 3 5 4

12. 2009.02.23 - SLIDE 12IS 240 – Spring 2009 Threshold =.2 2.4 3.4.2 4.3.3.3 5.1.4.4.1 1 2 3 4 2 0 3 0 1 4 0 0 0 5 1 0 0 1 1 2 3 4 2 1 3 5 4

13. 2009.02.23 - SLIDE 13IS 240 – Spring 2009 Threshold =.3 2.4 3.4.2 4.3.3.3 5.1.4.4.1 1 2 3 4 2 0 3 0 1 4 1 1 1 5 1 0 0 1 1 2 3 4 2 1 3 5 4

14. 2009.02.23 - SLIDE 14IS 240 – Spring 2009 K-means & Rocchio Clustering Agglomerative methods: Polythetic, Exclusive or Overlapping, Unordered clusters are order-dependent. Doc 1. Select initial centers (I.e. seed the space) 2. Assign docs to highest matching centers and compute centroids 3. Reassign all documents to centroid(s) Rocchio’s method

15. 2009.02.23 - SLIDE 15IS 240 – Spring 2009 Clustering Advantages: –See some main themes Disadvantage: –Many ways documents could group together are hidden Thinking point: what is the relationship to classification systems and facets?

16. 2009.02.23 - SLIDE 16IS 240 – Spring 2009 Automatic Class Assignment Doc Search Engine 1. Create pseudo-documents representing intellectually derived classes. 2. Search using document contents 3. Obtain ranked list 4. Assign document to N categories ranked over threshold. OR assign to top-ranked category Automatic Class Assignment: Polythetic, Exclusive or Overlapping, usually ordered clusters are order-independent, usually based on an intellectually derived scheme

17. 2009.02.23 - SLIDE 17IS 240 – Spring 2009 Automatic Categorization in Cheshire II Cheshire supports a method we call “classification clustering” that relies on having a set of records that have been indexed using some controlled vocabulary. Classification clustering has the following steps…

18. 2009.02.23 - SLIDE 18IS 240 – Spring 2009 Cheshire II - Cluster Generation Define basis for clustering records. –Select field (I.e., the contolled vocabulary terms) to form the basis of the cluster. –Evidence Fields to use as contents of the pseudo- documents. (E.g. the titles or other topical parts) During indexing cluster keys are generated with basis and evidence from each record. Cluster keys are sorted and merged on basis and pseudo-documents created for each unique basis element containing all evidence fields. Pseudo-Documents (Class clusters) are indexed on combined evidence fields.

19. 2009.02.23 - SLIDE 19IS 240 – Spring 2009 Cheshire II - Two-Stage Retrieval Using the LC Classification System –Pseudo-Document created for each LC class containing terms derived from “content-rich” portions of documents in that class (e.g., subject headings, titles, etc.) –Permits searching by any term in the class –Ranked Probabilistic retrieval techniques attempt to present the “Best Matches” to a query first. –User selects classes to feed back for the “second stage” search of documents. Can be used with any classified/Indexed collection.

20. 2009.02.23 - SLIDE 20IS 240 – Spring 2009 Problems with Vector Space There is no real theoretical basis for the assumption of a term space –it is more for visualization than having any real basis –most similarity measures work about the same regardless of model Terms are not really orthogonal dimensions –Terms are not independent of all other terms

21. 2009.02.23 - SLIDE 21IS 240 – Spring 2009 Today Probabilistic Models –Probabilistic Indexing (Model 1) –Probabilistic Retrieval (Model 2) –Unified Model (Model 3) –Model 0 and real-world IR –Regression Models –The “Okapi Weighting Formula”

22. 2009.02.23 - SLIDE 22IS 240 – Spring 2009 Probabilistic Models Rigorous formal model attempts to predict the probability that a given document will be relevant to a given query Ranks retrieved documents according to this probability of relevance (Probability Ranking Principle) Rely on accurate estimates of probabilities

23. 2009.02.23 - SLIDE 23IS 240 – Spring 2009 Probability Ranking Principle If a reference retrieval system’s response to each request is a ranking of the documents in the collections in the order of decreasing probability of usefulness to the user who submitted the request, where the probabilities are estimated as accurately as possible on the basis of whatever data has been made available to the system for this purpose, then the overall effectiveness of the system to its users will be the best that is obtainable on the basis of that data. Stephen E. Robertson, J. Documentation 1977

24. 2009.02.23 - SLIDE 24IS 240 – Spring 2009 Model 1 – Maron and Kuhns Concerned with estimating probabilities of relevance at the point of indexing: –If a patron came with a request using term t i, what is the probability that she/he would be satisfied with document D j ?

25. 2009.02.23 - SLIDE 25IS 240 – Spring 2009 Bayes’ Formula Bayesian statistical inference used in both models…

26. 2009.02.23 - SLIDE 26IS 240 – Spring 2009 Bayes’ theorem For example: A: disease B: symptom

27. 2009.02.23 - SLIDE 27IS 240 – Spring 2009 Bayes’ Theorem: Application Box1 Box2 p(box1) =.5 P(red ball | box1) =.4 P(blue ball | box1) =.6 p(box2) =.5 P(red ball | box2) =.5 P(blue ball | box2) =.5 Toss a fair coin. If it lands head up, draw a ball from box 1; otherwise, draw a ball from box 2. If the ball is blue, what is the probability that it is drawn from box 2?

28. 2009.02.23 - SLIDE 28IS 240 – Spring 2009 Bayes’ Theorem: Application in IR Goal: want to estimate the probability that a document D is relevant to a given query. It is easier to estimate log odds of probability of relevance

29. 2009.02.23 - SLIDE 29IS 240 – Spring 2009 Bayes’ Theorem: Application in IR If documents are represented by binary vectors, then Steven & Sparck Jones term weighting

30. 2009.02.23 - SLIDE 30IS 240 – Spring 2009 Bayes Theorem: Application in IR

31. 2009.02.23 - SLIDE 31IS 240 – Spring 2009 Bayes’ Theorem: Application in IR The task of estimating probability of relevance reduces to estimate the class-conditional probability density functions. Log odds of relevance:

32. 2009.02.23 - SLIDE 32IS 240 – Spring 2009 Model 1 A patron submits a query (call it Q) consisting of some specification of her/his information need. Different patrons submitting the same stated query may differ as to whether or not they judge a specific document to be relevant. The function of the retrieval system is to compute for each individual document the probability that it will be judged relevant by a patron who has submitted query Q. Robertson, Maron & Cooper, 1982

33. 2009.02.23 - SLIDE 33IS 240 – Spring 2009 Model 1 Bayes A is the class of events of using the system D i is the class of events of Document i being judged relevant I j is the class of queries consisting of the single term I j P(D i |A,I j ) = probability that if a query is submitted to the system then a relevant document is retrieved

34. 2009.02.23 - SLIDE 34IS 240 – Spring 2009 Model 2 Documents have many different properties; some documents have all the properties that the patron asked for, and other documents have only some or none of the properties. If the inquiring patron were to examine all of the documents in the collection she/he might find that some having all the sought after properties were relevant, but others (with the same properties) were not relevant. And conversely, he/she might find that some of the documents having none (or only a few) of the sought after properties were relevant, others not. The function of a document retrieval system is to compute the probability that a document is relevant, given that it has one (or a set) of specified properties. Robertson, Maron & Cooper, 1982

35. 2009.02.23 - SLIDE 35IS 240 – Spring 2009 Model 2 – Robertson & Sparck Jones Document Relevance Document indexing Given a term t and a query q + - + r n-r n - R-r N-n-R+r N-n R N-R N

36. 2009.02.23 - SLIDE 36IS 240 – Spring 2009 Robertson-Spark Jones Weights Retrospective formulation --

37. 2009.02.23 - SLIDE 37IS 240 – Spring 2009 Robertson-Sparck Jones Weights Predictive formulation

38. 2009.02.23 - SLIDE 38IS 240 – Spring 2009 Probabilistic Models: Some Unifying Notation D = All present and future documents Q = All present and future queries (D i,Q j ) = A document query pair x = class of similar documents, y = class of similar queries, Relevance is a relation:

39. 2009.02.23 - SLIDE 39IS 240 – Spring 2009 Probabilistic Models Model 1 -- Probabilistic Indexing, P(R|y,D i ) Model 2 -- Probabilistic Querying, P(R|Q j,x) Model 3 -- Merged Model, P(R| Q j, D i ) Model 0 -- P(R|y,x) Probabilities are estimated based on prior usage or relevance estimation

40. 2009.02.23 - SLIDE 40IS 240 – Spring 2009 Probabilistic Models Q D x y DiDi QjQj

41. 2009.02.23 - SLIDE 41IS 240 – Spring 2009 Logistic Regression Based on work by William Cooper, Fred Gey and Daniel Dabney. Builds a regression model for relevance prediction based on a set of training data Uses less restrictive independence assumptions than Model 2 –Linked Dependence

42. 2009.02.23 - SLIDE 42IS 240 – Spring 2009 Dependence assumptions In Model 2 term independence was assumed –P(R|A,B) = P(R|A)P(R|B) –This is not very realistic as we have discussed before Cooper, Gey, and Dabney proposed linked dependence: –If two or more retrieval clues are statistically dependent in the set of all relevance-related query- document pairs then they are statistically dependent to a corresponding degree in the set of all nonrelevance-related pairs. –Thus dependency in the relevant and nonrelevant documents is linked

43. 2009.02.23 - SLIDE 43IS 240 – Spring 2009 Linked Dependence Linked Dependence Assumption: there exists a positive real number K such that the following two conditions hold: –P(A,B|R) = K P(A|R) P(B|R) –When K=1 this is the same as binary independence

44. 2009.02.23 - SLIDE 44IS 240 – Spring 2009 Linked Dependence The Odds of an event E : O(E) = P(E)/P(E) (See paper for details) Multiplying by O(R) and taking logs we get:

45. 2009.02.23 - SLIDE 45IS 240 – Spring 2009 Probabilistic Models: Logistic Regression Estimates for relevance based on log- linear model with various statistical measures of document content as independent variables. Log odds of relevance is a linear function of attributes: Term contributions summed: Probability of Relevance is inverse of log odds:

46. 2009.02.23 - SLIDE 46IS 240 – Spring 2009 Logistic Regression 100 - 90 - 80 - 70 - 60 - 50 - 40 - 30 - 20 - 10 - 0 - 0 10 20 30 40 50 60 Term Frequency in Document Relevance

47. 2009.02.23 - SLIDE 47IS 240 – Spring 2009 Probabilistic Models: Logistic Regression attributes Average Absolute Query Frequency Query Length Average Absolute Document Frequency Document Length Average Inverse Document Frequency Inverse Document Frequency Number of Terms in common between query and document -- logged

48. 2009.02.23 - SLIDE 48IS 240 – Spring 2009 Probabilistic Models: Logistic Regression Probability of relevance is based on Logistic regression from a sample set of documents to determine values of the coefficients. At retrieval the probability estimate is obtained by: For the 6 X attribute measures shown previously

49. 2009.02.23 - SLIDE 49IS 240 – Spring 2009 Logistic Regression and Cheshire II The Cheshire II system uses Logistic Regression equations estimated from TREC full-text data In addition, an implementation of the Okapi BM-25 algorithm has been included also Demo (?)

50. 2009.02.23 - SLIDE 50IS 240 – Spring 2009 Current use of Probabilistic Models Most of the major systems in TREC now use the “Okapi BM-25 formula” (or Language Models -- more on those later) which incorporates the Robertson-Sparck Jones weights…

51. 2009.02.23 - SLIDE 51IS 240 – Spring 2009 Okapi BM-25 Where: Q is a query containing terms T K is k 1 ((1-b) + b.dl/avdl) k 1, b and k 3 are parameters, usually 1.2, 0.75 and 7-1000 tf is the frequency of the term in a specific document qtf is the frequency of the term in a topic from which Q was derived dl and avdl are the document length and the average document length measured in some convenient unit (e.g. bytes) w (1) is the Robertson-Sparck Jones weight.

52. 2009.02.23 - SLIDE 52IS 240 – Spring 2009 Probabilistic Models Strong theoretical basis In principle should supply the best predictions of relevance given available information Can be implemented similarly to Vector Relevance information is required -- or is “guestimated” Important indicators of relevance may not be term -- though terms only are usually used Optimally requires on- going collection of relevance information AdvantagesDisadvantages

53. 2009.02.23 - SLIDE 53IS 240 – Spring 2009 Vector and Probabilistic Models Support “natural language” queries Treat documents and queries the same Support relevance feedback searching Support ranked retrieval Differ primarily in theoretical basis and in how the ranking is calculated –Vector assumes relevance –Probabilistic relies on relevance judgments or estimates