(C) 2003, The University of Michigan1 Information Retrieval Handout #10 April 7, 2003

(C) 2003, The University of Michigan2 Course Information Instructor: Dragomir R. Radev Office: 3080, West Hall Connector Phone: (734) Office hours: M&F Course page: Class meets on Mondays, 1-4 PM in 409 West Hall

(C) 2003, The University of Michigan3 Schedule Final projects due 04/11 Final project presentations 04/14 Final exam 04/ essay questions, 2-3 problems

(C) 2003, The University of Michigan4 Language modeling

(C) 2003, The University of Michigan5 The problem In what order to show documents to the user?

(C) 2003, The University of Michigan6 Probabilistic retrieval Probabilistic retrieval [Robertson and Sparck Jones 1976] Given: query q and document d, estimate the probability that the user will find the document relevant. Assumption: the relevance depends only on the query and document representations

(C) 2003, The University of Michigan7 Probabilistic retrieval (cont’d) R = preferred set w i = index variables If P( R | d j ) > P ( R | d j ), then the document is relevant. prob. of randomly selecting d j from R can be ignored

(C) 2003, The University of Michigan8 Probabilistic retrieval (cont’d) Initial guess: Similarity:

(C) 2003, The University of Michigan9 Language models Each document generates a probability distribution Determine whether the query is from the same distribution as the document

(C) 2003, The University of Michigan10 Aspect models Hofmann Find diverse answers: e.g., queries about New Zealand from many perspectives

(C) 2003, The University of Michigan11 The Lemur system /clair4/projects/lemur-2.01/lemur/bin/ParseInQueryOp test2.sparam test2.squery /clair4/projects/lemur-2.01/lemur/bin/RetEval test2.param

(C) 2003, The University of Michigan12 The Lemur system more test2.squery #q1=#od2(New York); #q2=#od5(Spain Madrid); more test2.sparam outputFile = test2.query; /* result file */ more test2.query #ODN 2 LPAREN new york RPAREN #ODN 5 LPAREN spain madrid RPAREN more test2.out 1 AP AP LA AP AP AP AP AP WSJ SJMN AP AP WSJ AP WSJ WSJ WSJ