Presentation on theme: "Relevance Feedback and Query Expansion"— Presentation transcript:
1 Relevance Feedback and Query Expansion Debapriyo MajumdarInformation Retrieval – Spring 2015Indian Statistical Institute Kolkata
2 Importance of Recall Academic importance Not only of academic importanceUncertainty about availability of information: are the returned documents relevant at all?Query words may return small number of documents, none so relevantRelevance is not graded, but documents missed out could be more useful to the user in practiceWhat could have gone wrong?Many things, for instance …Some other choice of query words would have worked betterSearched for aircraft, results containing only plane were not returnedThe examples in the first bullet refer to the Cranfield data set which we have used as a programming exercise.
3 The gap between the user and the system A retrieval system tries to bridge this gapAssumption: the required information is present somewhereUser needs some informationThe gapThe retrieval system can only rely on the query words (in the simple setting)Wish: if the system could get another chance …
4 The gap between the user and the system A retrieval system tries to bridge this gapAssumption: the required information is present somewhereUser needs some informationIf the system gets another chanceModify the query to fill the gap betterUsually more query terms are added query expansionThe whole framework is called relevance feedback
5 Relevance Feedback User issues a query The system returns some results Sec. 9.1Relevance FeedbackUser issues a queryUsually short and simple queryThe system returns some resultsThe user marks some results as relevant or non-relevantThe system computes a better representation of the information need based on feedbackRelevance feedback can go through one or more iterations.It may be difficult to formulate a good query when you don’t know the collection well, so iterate
6 Example: similar pages Old time GoogleIf you (the user) tell me that this result is relevant, I can give you more such relevant documents
7 Example 2: Initial query/results SecExample 2: Initial query/resultsInitial query: New space satellite applications, 08/13/91, NASA Hasn’t Scrapped Imaging Spectrometer, 07/09/91, NASA Scratches Environment Gear From Satellite Plan, 04/04/90, Science Panel Backs NASA Satellite Plan, But Urges Launches of Smaller Probes, 09/09/91, A NASA Satellite Project Accomplishes Incredible Feat: Staying Within Budget, 07/24/90, Scientist Who Exposed Global Warming Proposes Satellites for Climate Research, 08/22/90, Report Provides Support for the Critics Of Using Big Satellites to Study Climate, 04/13/87, Arianespace Receives Satellite Launch Pact From Telesat Canada, 12/02/87, Telecommunications Tale of Two CompaniesUser then marks some relevant documents with “+”+++
8 Expanded query after relevance feedback SecExpanded query after relevance feedback2.074 new space satellite application nasa eos launch aster instrument arianespace bundespost ss rocket scientist broadcast earth oil measure
9 Results for expanded query SecResults for expanded query, 07/09/91, NASA Scratches Environment Gear From Satellite Plan , 08/13/91, NASA Hasn’t Scrapped Imaging Spectrometer , 08/07/89, When the Pentagon Launches a Secret Satellite, Space Sleuths Do Some Spy Work of Their Own , 07/31/89, NASA Uses ‘Warm’ Superconductors For Fast Circuit , 12/02/87, Telecommunications Tale of Two Companies , 07/09/91, Soviets May Adapt Parts of SS-20 Missile For Commercial Use , 07/12/88, Gaping Gap: Pentagon Lags in Race To Match the Soviets In Rocket Launchers , 06/14/90, Rescue of Satellite By Space Agency To Cost $90 Million218
10 The theoretically best query SecThe theoretically best queryThe information need is best “realized” by the relevant and non-relevant documentsxxxxoxxxxxxxxoxoxoxxooxxx non-relevant documentso relevant documentsOptimal query
11 SecKey concept: CentroidThe centroid is the center of mass of a set of pointsRecall that we represent documents as points in a high-dimensional spaceDefinition: Centroidwhere C is a set of documents.
12 SecRocchio AlgorithmThe Rocchio algorithm uses the vector space model to pick a relevance feedback queryRocchio seeks the query qopt that maximizesTries to separate docs marked relevant and non-relevantProblem: we don’t know the truly relevant docs
13 Rocchio Algorithm (SMART system) Used in practice:Dr = set of known relevant doc vectorsDnr = set of known irrelevant doc vectorsDifferent from Cr and Cnrqm = modified query vector; q0 = original query vector; α,β,γ: weights (hand-chosen or set empirically)New query moves toward relevant documents and away from irrelevant documentsTradeoff α vs. β/γ : If we have a lot of judged documents, we want a higher β/γ.Some weights in query vector can go negativeNegative term weights are ignored (set to 0)
14 Relevance feedback on initial query SecRelevance feedback on initial queryInitial queryxxxoxxxxxxxoxoxxoxooxxxxx known non-relevant documentso known relevant documentsRevised query
15 Relevance Feedback in vector spaces SecRelevance Feedback in vector spacesRelevance feedback can improve recall and precisionRelevance feedback is most useful for increasing recall in situations where recall is importantUsers can be expected to review results and to take time to iteratePositive feedback is more valuable than negative feedback (so, set < ; e.g. = 0.25, = 0.75).Many systems only allow positive feedback (=0).Just as we modified the query in the vector space model, we can also modify it here. I’m not aware of work that uses language model based Ir this way.
16 Relevance Feedback: Assumptions SecRelevance Feedback: AssumptionsA1: User has sufficient knowledge for initial query.A2: Relevance prototypes are “well-behaved”.Term distribution in relevant documents will be similarTerm distribution in non-relevant documents will be different from those in relevant documentsEither: All relevant documents are tightly clustered around a single prototype.Or: There are different prototypes, but they have significant vocabulary overlap.Similarities between relevant and irrelevant documents are small
17 Violation of A1 User does not have sufficient initial knowledge. SecViolation of A1User does not have sufficient initial knowledge.Examples:Misspellings (Brittany Speers).Cross-language information retrieval (hígado).Mismatch of searcher’s vocabulary vs. collection vocabularyCosmonaut/astronaut
18 Violation of A2 There are several relevance prototypes. Examples: SecViolation of A2There are several relevance prototypes.Examples:Burma/MyanmarContradictory government policiesPop stars that worked at Burger KingOften: instances of a general conceptGood editorial content can address problemReport on contradictory government policies
19 Evaluation of relevance feedback strategies SecEvaluation of relevance feedback strategiesUse q0 and compute precision and recall graphUse qm and compute precision recall graphAssess on all documents in the collectionSpectacular improvements, but … it’s cheating!Partly due to known relevant documents ranked higherMust evaluate with respect to documents not seen by userUse documents in residual collection (set of documents minus those assessed relevant)Measures usually then lower than for original queryBut a more realistic evaluationRelative performance can be validly comparedEmpirically, one round of relevance feedback is often very useful. Two rounds is sometimes marginally useful.It’s lower because the results aren’t on the documents that came in top on the original query, which are the ones most likely to be relevant.
20 Evaluation of relevance feedback SecEvaluation of relevance feedbackSecond method – assess only the docs not rated by the user in the first roundCould make relevance feedback look worse than it really isCan still assess relative performance of algorithmsMost satisfactory – use two collections each with their own relevance assessmentsq0 and user feedback from first collectionqm run on second collection and measured
21 SecEvaluation: CaveatTrue evaluation of usefulness must compare to other methods taking the same amount of time.Alternative to relevance feedback: User revises and resubmits query.Users may prefer revision/resubmission to having to judge relevance of documents.There is no clear evidence that relevance feedback is the “best use” of the user’s time.
22 Relevance Feedback: Problems Long queries are inefficient for typical IR engine.Long response times for user.High cost for retrieval system.Partial solution:Only reweight certain prominent termsPerhaps top 20 by term frequencyUsers are often reluctant to provide explicit feedbackIt’s often harder to understand why a particular document was retrieved after applying relevance feedbackA long vector space query is like a disjunction: you have to consider documents with any of the words, and sum partial cosine similarities over the various terms.
23 Relevance Feedback on the Web SecRelevance Feedback on the WebSome search engines offer a similar/related pages feature (a trivial form of relevance feedback)Google (link-based)AltavistaStanford WebBaseBut some don’t because it’s hard to explain to average user:AllthewebbingYahooExcite initially had true relevance feedback, but abandoned it due to lack of use.
24 Excite Relevance Feedback SecExcite Relevance FeedbackSpink et al. 2000Only about 4% of query sessions from a user used relevance feedback optionExpressed as “More like this” link next to each resultBut about 70% of users only looked at first page of results and didn’t pursue things furtherSo 4% is about 1/8 of people extending searchRelevance feedback improved results about 2/3 of the time
25 Pseudo relevance feedback SecPseudo relevance feedbackPseudo-relevance feedback automates the “manual” part of true relevance feedback.Pseudo-relevance algorithm:Retrieve a ranked list of hits for the user’s queryAssume that the top k documents are relevant.Do relevance feedback (e.g., Rocchio)Works very well on averageBut can go horribly wrong for some queries.Several iterations can cause query drift.Why?
26 SecQuery ExpansionIn relevance feedback, users give additional input (relevant/non-relevant) on documents, which is used to reweight terms in the documentsIn query expansion, users give additional input (good/bad search term) on words or phrases
27 Thesaurus-based query expansion SecThesaurus-based query expansionFor each term, t, in a query, expand the query with synonyms and related words of t from the thesaurusfeline → feline catMay weight added terms less than original query terms.Generally increases recallWidely used in many science/engineering fieldsMay significantly decrease precision, particularly with ambiguous terms.“interest rate” “interest rate fascinate evaluate”There is a high cost of manually producing a thesaurusAnd for updating it for scientific changesWe will study methods to build automatic thesaurus later in the course
28 Sources and Acknowledgements IR Book by Manning, Raghavan and Schuetze:Several slides are adapted from the slides by Prof. Nayak and Prof. Raghavan for their course in Stanford University