EFFICIENT COMPUTATION OF DIVERSE QUERY RESULTS Presenting: Karina Koifman Course : DB Seminar

Example

Yahoo! Autos

Maybe a better retrieval

Introduction  The article talks about the problem of efficiently computing diverse query results in online shopping applications.

The Goal  The goal of diverse query answering is to return a representative set of top-k answers from all the tuples that satisfy the user selection condition

 Users issues query for a product  Only most relevant answers are shown.  Many Duplications The Problem

 Existing Solutions  Definition of diversity  Impossibility results of diversity.  Query processing technique. Agenda

Existing Solutions Existing solutions are inefficient or do not work in all situations. Example:  Obtain all the query results and then pick a diverse subset from these results  doesn’t scale for large data sets.

Existing Solutions  Web search engines: first retrieve c × k and then pick a diverse subset from these.  It is more efficient than the previous method.  many duplicates  product sale. (inefficient and doesn’t guarantee diversity)

Existing Solutions  issuing multiple queries to obtain diverse results:

Pro’s\Con’s  The good:  Diversity  The Bad:  Hurts performance  Empty results *There are no Honda Accord convertibles

 Existing Solutions  Definition of diversity  Impossibility results of diversity.  Query processing technique. Agenda

 A diversity ordering of a relation R with attributes A, denoted by, is a total ordering of the attributes in A.  Example: Make ≺ Model ≺ Color ≺ Year ≺ Description ≺ Id Diversity Ordering

The DB example DescriptionYearColorModelMakeId Low miles2007GreenCivicHonda1 Low miles2007BlueCivicHonda2 Low miles2007RedCivicHonda3 Low miles2007BlackCivicHonda4 Low miles2006BlackCivicHonda5 Best Price2007BlueAccordHonda6 Good miles2006RedAccordHonda7 Rare2007GreenOdysseyHonda8 Good miles2006GreenOdysseyHonda9 Fun Car2007RedCRVHonda10 Good miles2006OrangeCRVHonda11 Low miles2007TanPriusToyota12 Low miles2007BlackCorollaToyota13 Low miles2007BlueTercelToyota14 Low miles2007BlueCamryToyota15

Similarity – SIM(X,Y) Low miles2007GreenCivicHonda1 Low miles2007BlueCivicHonda2 Low miles2007TanPriusToyota12 Low miles2007GreenCivicHonda1 Find a result set that minimizes

Example - Similarity DescriptionYearColorModelMakeId Low miles2007GreenCivicHonda1 Best Price2007BlueAccordHonda6 Rare2007GreenOdysseyHonda8 DescriptionYearColorModelMakeId Low miles2007GreenCivicHonda1 Low miles2007BlueCivicHonda2 Low miles2007TanPriusToyota12

Prefix DescriptionYearColorModelMakeId Low miles2007GreenCivicHonda1 DescriptionYearColorModelMakeId Low miles2007BlueCivicHonda2 DescriptionYearColorModelMakeId Rare2007GreenOdysseyHonda8 Good miles2006GreenOdysseyHonda9

Few more definitions  RES(R,Q) of size k  Given relation R and query Q, let maxval =

 Existing Solutions  Definition of diversity  Impossibility results of diversity.  Query processing technique. Agenda

Impossibility Results  Intuition: IR score of an item depends only on the item and possibly statistics from the entire corpus, but diversity depends on the other items in the query result set.

Inverted Lists Honda cars Honda Car d1d4d8d10d17d4d10d11d17d20 Merged Inverted List: d4d10d17

Impossibility Results  Item in an inverted list has a score, which can either be a global score (e.g., PageRank) or a value/keyword -dependent score (e.g., TF-IDF).  The items in each list are usually ordered by their score – so that we could handle top-k queries.  If we assume that we have a scoring function f() that is monotonic- which as a normal assumption for traditional IR system, then the article proofs either it’s not diverse or to inefficient\infeasible.

 Existing Solutions  Definition of diversity  Impossibility results of diversity.  Query processing technique. Agenda

The DB example DescriptionYearColorModelMakeId Low miles2007GreenCivicHonda1 Low miles2007BlueCivicHonda2 Low miles2007RedCivicHonda3 Low miles2007BlackCivicHonda4 Low miles2006BlackCivicHonda5 Best Price2007BlueAccordHonda6 Good miles2006RedAccordHonda7 Rare2007GreenOdysseyHonda8 Good miles2006GreenOdysseyHonda9 Fun Car2007RedCRVHonda10 Good miles2006OrangeCRVHonda11 Low miles2007TanPriusToyota12 Low miles2007BlackCorollaToyota13 Low miles2007BlueTercelToyota14 Low miles2007BlueCamryToyota15

The car indexing example

One-pass Algorithm Lets say Q looks for descriptions with ‘Low’, with k=3 Honda.Civic.Green.2007.’Low miles’ Pick first K Initialization go to next option and check if better, if so – prune While we can improve Diversity

One-pass Algorithm We start from two Civics, then we know that we need only one more so we pick the next Civic

One-pass Algorithm Then we look for another in next level (Accord)- no such, because it doesn’t have ‘Low’ in it (also no other in that level).

One-pass Algorithm Then we look for another in next level (make)- and prune, This is maximum diverse – we stop here.

One-pass Algorithm If we had a Ford, we would continue Ford Focus 0 Black Low miles 0

Scored One-pass Algorithm Give each car a score, then the query would take this score as parameter- minScore- smallest score in the result set, Choose next next ID by : The smallest ID such that score(id)>=root.minScore. And the algorithm proceeds as before.

Probing Algorithm Main idea: to go over all the cars as they were on an axis K=1 K=2 K=3

Advantage of bidirectional exploring  “Honda” only has one child, we found it quickly not exploring every option (only civic).  Each time we add a node to the diverse solution we do not have to prune it- unlike the OnePass algorithm.

WAND algorithm  WAND is an efficient method of obtaining top-K lists of scored results, without explicitly merging the full inverted lists.  AND(X1,X2,...Xk) ≡ WAND(X1,1,X2,1,...Xk,1,k),  OR(X1,X2,...Xk) ≡ WAND(X1,1,X2,1,...Xk,1,1).  To obtain k best results the operator uses the upper bounds of maximum contribution, and temp threshold. WAND(X1,UB1,X2,UB2,...,Xk,UBk, θ )

Scored Probing Algorithm We use the WAND algorithm- to obtain the top-k list. Next step is marking all possible nodes to add- as MIDDLE. we also maintain a heap – for a node with minimum child. Each step we move nodes from tentative to useful.

Experiments MultQ – rewriting the query as multiple queries and merging their results. Naïve – all the results of a query Basic - just first k answers – without diversity. OnePass, Probe – our algorithms U = unscored S = scored

Experiments

Conclusions  Formalized diversity in structured search and proposed inverted-list algorithms.  The experiments showed that the algorithms are scalable and efficient.  In particular, diversity can be implemented with little additional overhead when compared to traditional approaches

Extension of the algorithm  Assign higher weights to Hondas and Toyotas when compared to Teslas, so that the diverse results have more Hondas and Toyotas.

Questions? Thank You !