Tuning Before Feedback: Combining Ranking Discovery and Blind Feedback for Robust Retrieval* Weiguo Fan, Ming Luo, Li Wang, Wensi Xi, and Edward A. Fox Digital Library Research Laboratory, Virginia Tech *This research is supported by the National Science Foundation under Grant Numbers IIS , DUE and DUE

Outline Introduction Research Questions Approach: Ranking Tuning + Blind Fdbk Experiment Results Conclusion

Introduction Ranking functions play an important role in IR performance Blind feedback (pseudo-relevance feedback) has been found very useful for ad hoc retrieval Why not combine ranking function optimization with blind feedback to improve robustness?

Research Questions Does blind feedback work even better on fine- tuned ranking functions as compared to on traditional ranking functions such as Okapi BM25? Does the type of query (very short vs. very long) have any impact on the combination approach? Can the ranking function discovered, in combination with blind feedback, extrapolate well for new unseen queries?

Our Approach Use ARRANGER a Genetic Programming-based discovery engine to perform the ranking function tuning [Fan 2003tkde, Fan 2004ip&m, Fan 2004jasist] Combine ranking tuning and feedback Test on different types of queries

RF Discovery Problem Feedback Training Data Input Ranking Function Discovery Ranking Function f Output

Ranking Function Optimization Ranking Function Tuning is an art! – Paul Kantor Why not adaptively discover RF by Genetic Programming? Huge search space Discrete objective function Modeling advantage What is GP? Problem solving system designed based on principles of evolution and heredity Widely used for structure discovery, functional form discovery, other data mining and optimization tasks

Genetic Algorithms/Programming Representation: Vector of bit strings or real numbers for GA Complex data structures: trees, arrays for GP Genetic transformation Reproduction Crossover Mutation IR application [Gordon’88, ’91], [Chen’98a, ’98b], [Pathak’00], etc.

Essential GP Components ComponentsMeaning TerminalsLeaf nodes in the tree structure (i.e., x, y). FunctionsNon-leaf nodes used to combine the leaf nodes. Commonly, numerical operations: +, -, *, /, log, sqrt. Fitness function The objective function GP aims to optimize. ReproductionA genetic operator that copies the individuals with the best fitness values directly into the population of the next generation without going through the crossover operation. CrossoverA genetic operator aiming to improve the diversity as well as the genetic fitness of the population. See details in next slide.

Example of Crossover in GP tf*(tf+df) tf*(N/df) + df * tf * + df Crossover Parent1 Parent2 Child1 Child2 N/df+df (tf*df)+df N / df tf + Generation: N Generation: N+1 N / df tf +

The ARRANGER Engine 1.Split the training data into training and validation 2.Generate an initial population of random “ranking functions” 3.Evaluate the fitness of each “ranking function” in the population and record 10 best ones 4.If stopping criteria is not met, generate the next generation of population by genetic transformation, go to Step 3. 5.Validate the recorded best “ranking functions” and select the best one as the RF Start Initialize Population Evaluate Fitness Apply Crossover Stop? Validate and Output End

The ARRANGER Engine 1.Split the training data into training and validation 2.Generate an initial population of random “ranking functions” 3.Evaluate the fitness of each “ranking function” in the population and record 10 best ones 4.If stopping criteria is not met, generate the next generation of population by genetic transformation, go to Step 3. 5.Validate the recorded best “ranking functions” and select the best one as the RF

The ARRANGER Engine Start Initialize Population Evaluate Fitness Apply Crossover Stop? Validate and Output End

Blind Feedback Automatically adds more terms to a user’s query to enhance the performance of search engines by assuming top ranked docs relevant Some examples Rocchio (performs best in our experiment) Dec-Hi Kullback-Leibler Divergence (KLD) Chi-Square

An Integrated Model

Experiment Setting Data 2003 Robust Track data (from TREC 6, 7, 8) Training Queries 150 old queries from TREC 6, 7, 8 Test Questions 50 very hard queries + 50 new queries

The Results on 150 Training Queries Run No.DescShort Okapi without BF (Baseline) Okapi with BF (+10.4%) (+8.7%) RF 1 without BF (+15.6%) (+9.1%) RF 1 with BF (+28.8%) (+21.3%)

Results on Test Queries (1)

Results on Test Queries (2)

Conclusions Blind feedback works well on GP trained queries. Ranking function combined with blind feedback works with new queries Two stage model responds differently to Desc query (slightly better) and Long query

Thank You! Q&A?