Presentation is loading. Please wait.

Presentation is loading. Please wait.

Sreenivas Gollapudi Search Labs, Microsoft Research Joint work with Aneesh Sharma (Stanford), Samuel Ieong, Alan Halverson, and Rakesh Agrawal (Microsoft.

Similar presentations


Presentation on theme: "Sreenivas Gollapudi Search Labs, Microsoft Research Joint work with Aneesh Sharma (Stanford), Samuel Ieong, Alan Halverson, and Rakesh Agrawal (Microsoft."— Presentation transcript:

1 Sreenivas Gollapudi Search Labs, Microsoft Research Joint work with Aneesh Sharma (Stanford), Samuel Ieong, Alan Halverson, and Rakesh Agrawal (Microsoft Research)

2 wine 2009

3 Intuitive definition Represent a variety of relevant meanings for a given query Mathematical definitions: Minimizing query abandonment Want to represent different user categories Trade-off between relevance and novelty

4 Query and document similarities Maximal Marginal Relevance [CG98] Personalized re-ranking of results [RD06] Probability Ranking Principle not optimal [CK06] Query abandonment Topical diversification [Z+05, AGHI09] Needs topical (categorical) information Loss minimization framework [Z02, ZL06] Diminishing returns for docs w/ the same intent is a specific loss function [AGHI09]

5 Express diversity requirements in terms of desired properties Define objectives that satisfy these properties Develop efficient algorithms Metrics and evaluation methodologies

6 Inspired by similar approaches for Recommendation systems [Andersen et al 08] Ranking [Altman, Tennenholtz 07] Clustering [Kleinberg 02] Map the space of functions – a basis vector

7 Input: Candidate documents: U={u 1,u 2,…, u n }, query q Relevance function: w q (u i ) Distance function: d q (u i, u j ) (symmetric, non-metric) Size k of output result set w q (u 5 ) u5u5 u1u1 u2u2 u3u3 u4u4 u6u6 d q (,u 2,u 4 )

8 Output Diversified set S* of documents (|S*|= k) Diversification function: f : S x w q x d q R + S* = argmax f(S) (|S|=k) u5u5 u1u1 u2u2 u3u3 u4u4 u6u6 k = 3 S* = {u 1,u 2,u 6 }

9 1. Scale-invariance 2. Consistency 3. Richness 4. Strength a)Relevance b)Diversity 5. Stability 6. Two technical properties

10 S* = argmax S f(S, w(·), d(·, ·)) = argmax S f(S, w΄(·), d΄(·, ·)) w΄(u i ) = α · w(u i ) d΄(u i,u j ) = α · d(u i,u j ) No built-in scale for f ! S*(3)

11 S* = argmax S f(S, w(·), d(·, ·)) = argmax S f(S, w΄(·), d΄(·, ·)) w΄(u i ) = w(u i ) + a i for u i є S* d΄(u i,u j ) = d(u i,u j ) + b i for u i and/or u j є S* Increasing relevance/ diversity doesnt hurt! S*(3)

12 S*(k) = argmax S f(S, w(·), d(·, ·),k) S*(k) S*(k+1) for all k Output set shouldnt oscillate (change arbitrarily) with size S*(3) S*(4)

13 Proof via constructive argument Theorem: No function f can satisfy all the axioms simultaneously. Scale-invariance, Consistency, Richness, Strength of Relevance/Diversity, Stability, Two technical properties

14 Baseline for what is possible Mathematical criteria for choosing f Modular approach: f is independent of specific w q (·) and d q (·, ·)!

15 Express diversity requirements in terms of desired properties Define objectives that satisfy these properties Develop efficient algorithms Metrics and evaluation methodologies

16 Input: U={u 1,u 2,…,u n }, k, w q (·) and d q (·, ·) Some set of (top) n results Output: S* = argmax S f(S, w(·), d(·, ·),k) Find the most diverse set of results of size k Advantages: Can integrate f with existing ranking engine Modular, plug-and-play framework

17 Max-sum (avg) objective: u5u5 u1u1 u2u2 u3u3 u4u4 u6u6 k = 3 S* = {u 1,u 2,u 6 } Violates stability! u3u3 u5u5 k = 4 S* = {u 1,u 3,u 5,u 6 }

18 Max-min objective: u5u5 u1u1 u2u2 u3u3 u4u4 u6u6 k = 3 S* = {u 1,u 2,u 6 } Violates consistency and stability! u5u5 S* = {u 1,u 5,u 6 }

19 A taxonomy-based diversification objective Uses the analogy of marginal utility to determine whether to include more results from an already covered category Violates stability and one of the technical axioms

20 Express diversity requirements in terms of desired properties Define objectives that satisfy these properties Develop efficient algorithms Metrics and evaluation methodologies

21 Recast as facility dispersion Max-sum (MaxSumDispersion): Max-min (MaxMinDispersion): Known approximation algorithms Lower bounds Lots of other facility dispersion objectives and algorithms

22 S = c C, U (c |q) P (c |q) while |S| < k do for d D do g (d |q, c) c U (c |q)V (d |q,c) end for d argmax g (d | q, c) S S {d } c C, U (c |q) (1V (d |q, c))U (c |q) D D \ {d } end while P(c | q): conditional prob of intent c given query q g(d | q, c): current prob of d satisfying q, c Update the utility of a category

23 Intent distribution: P (R |q) = 0.8, P (B |q) = DV(d | q, c) g(d | q, c) U(R | q) =U(B | q) = × 0.8 × 0.2 × 0.08 × × 0.08 × S Actually produces an ordered set of results Results not proportional to intent distribution Results not according to (raw) quality Better results less needed to be shown

24 Express diversity requirements in terms of desired properties Define objectives that satisfy these properties Develop efficient algorithms Metrics and evaluation methodologies

25 Approach Represent real queries Scale beyond a few user studies Problem: Hard to define ground truth Use disambiguated information sources on the web as the ground truth Incorporate intent into human judgments Can exploit the user distribution (need to be careful)

26 Query = Wikipedia disambiguation page title Large-scale ground truth set Open source Growing in size

27 Novelty Coverage of wikipedia topics Relevance coverage of top Wikipedia results

28 Relevance function: 1/position Can use the search engine score Maybe use query category information Distance function: Compute TF-IDF distances Jaccard similarity score for two documents A and B:

29 Topics/categories = list of disambiguation topics Given a set S k of results: For each result, compute a distribution over topics (using our d(·, ·)) Sum confidence over all topics Threshold to get # topics represented jaguar.com Jaguar cat (0.1) Jaguar car (0.9) wikipedia.org/jaguar Jaguar cat (0.8) Jaguar car (0.2) Category confidence Jaguar cat: Jaguar car: Threshold = 1.0 Jaguar cat: 0 Jaguar car: 1

30 Query – get ranking of search restricted to Wikipedia pages a(i) = position of Wikipedia topic i in this list b(i) = position of Wikipedia topic i in list being tested Relevance is measured in terms of reciprocal ranks:

31 Take expectation over distribution of intents Interpretation: how will the average user feel? Consider Classic: NDCG-IA depends on intent distribution and intent-specific NDCG

32 Created two types of HITs on Mechanical Turk Query classification: workers are asked to choose among three interpretations Document rating (under the given interpretation) Two additional evaluations MT classification + current ratings MT classification + MT document ratings

33 When is it right to diversify? Users have certain expectations about the workings of a search engine What is the best way to diversify? Evaluate approaches beyond diversifying the retrieved results Metrics that capture both relevance and diversity Some preliminary work suggests that there will be certain trade-offs to make

34 Questions?

35 Otherwise, need to encode explicit user model in the metric Selection only needs k (which is 10) Later, can rank set according to relevance Personalize based on clicks Alternative to stability: Select sets repeatedly (this loses information) Could refine selection online, based on user clicks

36

37

38 Results for query cd player Relevance: popularity Distance: from product hierarchy

39

40

41 Many metrics for relevance Normalized discounted cumulative gains at k Mean average precision at k Mean reciprocal rank (MRR) Some metrics for diversity Maximal marginal relevance (MMR) [CG98] Nugget-based instantiation of NDCG [C+08] Want a metric that can take into account both relevance and diversity [JK00]

42 D IVERSIFY ( K ) Given a query q, a set of documents D, distribution P(c | q), quality estimates V(d | c, q), and integer k Find a set of docs S D with |S| = k that maximizes interpreted as the probability that the set S is relevant to the query over all possible intentions Find at least one relevant docMultiple intents

43 Makes explicit use of taxonomy In contrast, similarity-based: [CG98], [CK06], [RKJ08] Captures both diversification and doc relevance In contrast, coverage-based: [Z+05], [C+08], [V+08] Specific form of loss minimization [Z02], [ZL06] Diminishing returns for docs w/ the same intent Objective is order-independent Assumes that all users read k results May want to optimize k P(k) P(S | q)


Download ppt "Sreenivas Gollapudi Search Labs, Microsoft Research Joint work with Aneesh Sharma (Stanford), Samuel Ieong, Alan Halverson, and Rakesh Agrawal (Microsoft."

Similar presentations


Ads by Google