Seesaw Personalized Web Search Jaime Teevan, MIT with Susan T. Dumais and Eric Horvitz, MSR

Query expansion Personalization Algorithms Standard IR Document Query User Server Client

Query expansion Personalization Algorithms Standard IR Document Query User Server Client v. Result re-ranking

Result Re-Ranking Ensures privacy Good evaluation framework Can look at rich user profile Look at light weight user models  Collected on server side  Sent as query expansion

Seesaw Search EngineSeesaw dog 1 cat10 india 2 mit 4 search93 amherst12 vegas 1

Seesaw Search Engine query dog 1 cat10 india 2 mit 4 search93 amherst12 vegas 1

Seesaw Search Engine query dog 1 cat10 india 2 mit 4 search93 amherst12 vegas 1 dog cat monkey banana food baby infant child boy girl forest hiking walking gorp baby infant child boy girl csail mit artificial research robot web search retrieval ir hunt

Seesaw Search Engine query dog 1 cat10 india 2 mit 4 search93 amherst12 vegas Search results page web search retrieval ir hunt 1.3

Calculating a Document’s Score Based on standard tf.idf web search retrieval ir hunt 1.3

Calculating a Document’s Score Based on standard tf.idf (r i +0.5)(N-n i -R+r i +0.5) (n i -r i +0.5)(R-r i +0.5) w i = log User as relevance feedback  Stuff I’ve Seen index  More is better

Finding the Score Efficiently Corpus representation (N, n i )  Web statistics  Result set Document representation  Download document  Use result set snippet Efficiency hacks generally OK!

Evaluating Personalized Search 15 evaluators Evaluate 50 results for a query  Highly relevant  Relevant  Irrelevant Measure algorithm quality  DCG(i) = { Gain(i), DCG (i–1) + Gain(i)/log(i), if i = 1 otherwise

Evaluating Personalized Search Query selection  Chose from 10 pre-selected queries  Previously issued query cancer Microsoft traffic … bison frise Red Sox airlines … Las Vegas rice McDonalds … Pre-selected 53 pre-selected (2-9/query) Total: 137 Joe Mary

Seesaw Improves Text Retrieval Random Relevance Feedback Seesaw

Text Features Not Enough

Take Advantage of Web Ranking

Further Exploration Explore larger parameter space Learn parameters  Based on individual  Based on query  Based on results Give user control?

Making Seesaw Practical Learn most about personalization by deploying a system Best algorithm reasonably efficient Merging server and client  Query expansion Get more relevant results in the set to be re-ranked  Design snippets for personalization

User Interface Issues Make personalization transparent Give user control over personalization  Slider between Web and personalized results  Allows for background computation Creates problem with re-finding  Results change as user model changes  Thesis research – Re:Search Engine

Thank you!

END

Personalizing Web Search Motivation Algorithms Results Future Work

Personalizing Web Search Motivation Algorithms Results Future Work

Study of Personal Relevancy 15 participants  Microsoft employees  Managers, support staff, programmers, … Evaluate 50 results for a query  Highly relevant  Relevant  Irrelevant ~10 queries per person

Study of Personal Relevancy Query selection  Chose from 10 pre-selected queries  Previously issued query cancer Microsoft traffic … bison frise Red Sox airlines … Las Vegas rice McDonalds … Pre-selected 53 pre-selected (2-9/query) Total: 137 Joe Mary

Relevant Results Have Low Rank Highly Relevant Relevant Irrelevant

Relevant Results Have Low Rank Highly Relevant Relevant Irrelevant Rater 1 Rater 2

Same Results Rated Differently Average inter-rater reliability: 56% Different from previous research  Belkin: 94% IRR in TREC  Eastman: 85% IRR on the Web Asked for personal relevance judgments Some queries more correlated than others

Same Query, Different Intent Different meanings  “Information about the astronomical/astrological sign of cancer”  “information about cancer treatments” Different intents  “is there any new tests for cancer?”  “information about cancer treatments”

Same Intent, Different Evaluation Query: Microsoft  “information about microsoft, the company”  “Things related to the Microsoft corporation”  “Information on Microsoft Corp” 31/50 rated as not irrelevant  Only 6/31 do more than one agree  All three agree only for  Inter-rater reliability: 56%

Search Engines are for the Masses JoeMary

Much Room for Improvement Group ranking  Best improves on Web by 38%  More people  Less improvement

Much Room for Improvement Group ranking  Best improves on Web by 38%  More people  Less improvement Personal ranking  Best improves on Web by 55%  Remains constant

- Seesaw Search Engine- See- Seesaw Personalizing Web Search Motivation Algorithms Results Future Work

BM25 N nini NniNni w i = log riri R with Relevance Feedback Score = Σ tf i * w i

N nini (r i +0.5)(N-n i -R+r i +0.5) (n i -r i +0.5)(R-r i +0.5) riri R w i = log Score = Σ tf i * w i BM25with Relevance Feedback

(r i +0.5)(N-n i -R+r i +0.5) (n i - r i +0.5)(R-r i +0.5) User Model as Relevance Feedback N nini R riri Score = Σ tf i * w i (r i +0.5)(N’-n i ’-R+r i +0.5) (n i ’- r i +0.5)(R-r i +0.5) w i = log N’ = N+R n i ’ = n i +ri

User Model as Relevance Feedback N nini R riri World User Score = Σ tf i * w i

User Model as Relevance Feedback R riri User N nini World World related to query N nini Score = Σ tf i * w i

User Model as Relevance Feedback N nini R riri World User World related to query User related to query R N nini riri Query Focused Matching Score = Σ tf i * w i

User Model as Relevance Feedback N nini R riri World User Web related to query User related to query R N riri Query Focused Matching nini World Focused Matching Score = Σ tf i * w i

Parameters Matching User representation World representation Query expansion

Parameters Matching User representation World representation Query expansion Query focused World focused

Parameters Matching User representation World representation Query expansion Query focused World focused

User Representation Stuff I’ve Seen (SIS) index  MSR research project [Dumais, et al.]  Index of everything a user’s seen Recently indexed documents Web documents in SIS index Query history None

Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history None

Parameters Matching User representation World representation Query expansion Query Focused World Focused All SIS Recent SIS Web SIS Query History None

World Representation Document Representation  Full text  Title and snippet Corpus Representation  Web  Result set – title and snippet  Result set – full text

Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history None Full text Title and snippet Web Result set – full text Result set – title and snippet

Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history None Full text Title and snippet Web Result set – full text Result set – title and snippet

Query Expansion All words in document Query focused The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through...

Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

Parameters Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

Personalizing Web Search Motivation Algorithms Results Future Work

Best Parameter Settings Matching User representation World representation Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused All SIS Recent SIS Web SIS All SIS Recent SIS Web SIS Query history None Full text All words Query focused World focused Result set – title and snippet Web Query focused All SIS Title and snippet Result set – title and snippet Query focused

Seesaw Improves Retrieval No user model Random Relevance Feedback Seesaw

Text Alone Not Enough

Incorporate Non-text Features

Summary Rich user model important for search personalization Seesaw improves text based retrieval Need other features to improve Web Lots of room for improvement future

Personalizing Web Search Motivation Algorithms Results Future Work  Further exploration  Making Seesaw practical  User interface issues

Further Exploration Explore larger parameter space Learn parameters  Based on individual  Based on query  Based on results Give user control?

Making Seesaw Practical Learn most about personalization by deploying a system Best algorithm reasonably efficient Merging server and client  Query expansion Get more relevant results in the set to be re-ranked  Design snippets for personalization

User Interface Issues Make personalization transparent Give user control over personalization  Slider between Web and personalized results  Allows for background computation Creates problem with re-finding  Results change as user model changes  Thesis research – Re:Search Engine

Thank you!

Search Engines are for the Masses Best common ranking  DCG(i) = {  Sort results by number marked highly relevant, then by relevant Measure distance with Kendall-Tau Web ranking more similar to common  Individual’s ranking distance:  Common ranking distance: Gain(i),if i = 1 DCG(i–1) + Gain(i)/log(i),otherwise