Presentation is loading. Please wait.

Presentation is loading. Please wait.

2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 1 龙星计划课程 : 信息检索 Personalized Search & User Modeling ChengXiang Zhai.

Similar presentations


Presentation on theme: "2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 1 龙星计划课程 : 信息检索 Personalized Search & User Modeling ChengXiang Zhai."— Presentation transcript:

1 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 龙星计划课程 : 信息检索 Personalized Search & User Modeling ChengXiang Zhai ( 翟成祥 ) Department of Computer Science Graduate School of Library & Information Science Institute for Genomic Biology, Statistics University of Illinois, Urbana-Champaign

2 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What is Personalized Search? Use more user information than the user’s query in retrieval –“more information” = user’s interaction history  Implicit feedback –“more information” = user’s judgments or user’s answer to clarification questions  explicit feedback Personalization can be done in multiple ways: –Personalize the collection –Personalize ranking –Personalize result presentation –… Personalized search = user modeling + model exploitation

3 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Why Personalized Search? The more we know about the user’s information need, the more likely we can get relevant documents, thus we should know as much as we can about the users When a query doesn’t work well, personalized search would be extremely helpful.

4 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Client-Side vs. Server-Side Personalization Server-Side (most work, including commercial products): –Sees global information (all documents, all users) – Limited user information (can’t see activities outside search results) –Privacy issue Client-Side (UCAIR): –More information about the user, thus more accurate user modeling (complete interaction history + other user activities) –More scalable (“distributed personalization”) –Alleviate the problem of privacy Combination of server-side and client-side? How?

5 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Outline A framework for optimal interactive retrieval Implicit feedback (no user effort) –Within a search session –For improving result organization Explicit feedback (with user effort) –Term feedback –Active feedback Improving search result organization

6 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, A Framework for Optimal Interactive Retrieval [Shen et al. 05]

7 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, IR as Sequential Decision Making UserSystem A 1 : Enter a query Which documents to present? How to present them? R i : results (i=1, 2, 3, …) Which documents to view? A 2 : View document Which part of the document to show? How? R’: Document content View more? A 3 : Click on “Back” button (Information Need) (Model of Information Need)

8 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Retrieval Decisions User U: A 1 A 2 … … A t-1 A t System: R 1 R 2 … … R t-1 Given U, C, A t, and H, choose the best R t from all possible responses to A t History H={(A i,R i )} i=1, …, t-1 Document Collection C Query=“Jaguar” All possible rankings of C The best ranking for the query Click on “Next” button All possible rankings of unseen docs The best ranking of unseen docs R t  r(A t ) R t =?

9 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, A Risk Minimization Framework User: U Interaction history: H Current user action: A t Document collection: C Observed All possible responses: r(A t )={r 1, …, r n } User Model M=(S,  U …) Seen docs Information need L(r i,A t,M)Loss Function Optimal response: r* (minimum loss) ObservedInferred Bayes risk

10 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Approximate the Bayes risk by the loss at the mode of the posterior distribution Two-step procedure –Step 1: Compute an updated user model M* based on the currently available information –Step 2: Given M*, choose a response to minimize the loss function A Simplified Two-Step Decision-Making Procedure

11 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Optimal Interactive Retrieval User A1A1 UC M* 1 P(M 1 |U,H,A 1,C) L(r,A 1,M* 1 ) R1R1 A2A2 L(r,A 2,M* 2 ) R2R2 M* 2 P(M 2 |U,H,A 2,C) A3A3 … Collection IR system

12 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Refinement of Risk Minimization r(A t ): decision space (A t dependent) –r(A t ) = all possible subsets of C (document selection) –r(A t ) = all possible rankings of docs in C –r(A t ) = all possible rankings of unseen docs –r(A t ) = all possible subsets of C + summarization strategies M: user model –Essential component:  U = user information need –S = seen documents –n = “Topic is new to the user” L(R t,A t,M): loss function –Generally measures the utility of R t for a user modeled as M –Often encodes retrieval criteria (e.g., using M to select a ranking of docs) P(M|U, H, A t, C): user model inference –Often involves estimating a unigram language model  U

13 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Case 1: Context-Insensitive IR –A t =“enter a query Q” –r(A t ) = all possible rankings of docs in C –M=  U, unigram language model (word distribution) –p(M|U,H,At,C)=p(  U |Q)

14 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Case 2: Implicit Feedback –A t =“enter a query Q” –r(A t ) = all possible rankings of docs in C –M=  U, unigram language model (word distribution) –H={previous queries} + {viewed snippets} –p(M|U,H,At,C)=p(  U |Q,H)

15 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Case 3: General Implicit Feedback –A t =“enter a query Q” or “Back” button, “Next” button –r(A t ) = all possible rankings of unseen docs in C –M= (  U, S), S= seen documents –H={previous queries} + {viewed snippets} –p(M|U,H,At,C)=p(  U |Q,H)

16 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Case 4: User-Specific Result Summary –A t =“enter a query Q” –r(A t ) = {(D,  )}, D  C, |D|=k,  {“snippet”,”overview”} –M= (  U, n), n  {0,1} “topic is new to the user” –p(M|U,H,At,C)=p(  U,n|Q,H), M*=(  *, n*) n*=1n*=0  i =snippet 10  i =overview 01 Choose k most relevant docs If a new topic (n*=1), give an overview summary; otherwise, a regular snippet summary

17 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What You Should Know Disadvantages and advantages of client-side vs. server-side personalization The optimal interactive retrieval framework provides a general way to model personalized search –Maximum user modeling –Immediate benefit (“eager feedback”) Personalization can be potentially done for all the components and steps in a retrieval system

18 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Implicit Feedback [Shen et al. 05, Tan et al. 06]

19 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, “Jaguar” Example Car Software Animal Suppose we know: 1.Previous query = “racing cars” vs. “Apple OS” 2.“car” occurs far more frequently than “Apple” in pages browsed by the user in the last 20 days 3. User just viewed an “Apple OS” document

20 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, How can we exploit such implicit feedback information that already naturally exists to improve ranking accuracy?

21 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Risk Minimization for Implicit Feedback –A t =“enter a query Q” –r(A t ) = all possible rankings of docs in C –M=  U, unigram language model (word distribution) –H={previous queries} + {viewed snippets} –p(M|U,H,At,C)=p(  U |Q,H) Need to estimate a context-sensitive LM

22 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Scenario 1: Use Information in one Session [Shen et al. 05] Q2Q2 C 2 ={C 2,1, C 2,2,C 2,3, … } … C 1 ={C 1,1, C 1,2,C 1,3, …} User Clickthrough QkQk Q1Q1 User Query e.g., Apple software e.g., Apple - Mac OS X Apple - Mac OS X The Apple Mac OS X product page. Describes features in the current version of Mac OS X, … e.g., Jaguar User Model: Query HistoryClickthrough

23 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Method1: Fixed Coeff. Interpolation (FixInt) QkQk Q1Q1 Q k-1 … C1C1 C k-1 … Average user query history and clickthrough Linearly interpolate history models Linearly interpolate current query and history model

24 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Method 2: Bayesian Interpolation (BayesInt) Q1Q1 Q k-1 … C1C1 C k-1 … Average user query and clickthrough history Intuition: trust the current query Q k more if it’s longer QkQk Dirichlet Prior

25 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Method 3: Online Bayesian Updating (OnlineUp) QkQk C2C2 Q1Q1 Intuition: incremental updating of the language model C1C1 Q2Q2

26 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Method 4: Batch Bayesian Update (BatchUp) C2C2 … C k-1 Intuition: all clickthrough data are equally useful QkQk Q1Q1 C1C1 Q2Q2

27 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, TREC Style Evaluation Data collection: TREC AP88-90 Topics: 30 hard topics of TREC topics System: search engine + RDBMS Context: Query and clickthrough history of 3 participants ( )

28 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Example of a Hard Topic 2 (283 relevant docs in documents) Acquisitions Document discusses a currently proposed acquisition involving a U.S. company and a foreign company. To be relevant, a document must discuss a currently proposed acquisition (which may or may not be identified by type, e.g., merger, buyout, leveraged buyout, hostile takeover, friendly acquisition). The suitor and target must be identified by name; the nationality of one of the companies must be identified as U.S. and the nationality of the other company must be identified as NOT U.S.

29 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Performance of the Hard Topic Q1: acquisition u.s. foreign company MAP: 0.004; Q2: acquisition merge takeover u.s. foreign company MAP: 0.026; Q3: acquire merge foreign abroad international MAP: 0.004; Q4: acquire merge takeover foreign european japan MAP: 0.027; 0.200

30 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Overall Effect of Search Context Query FixInt (  =0.1,  =1.0) BayesInt (  =0.2, =5.0) OnlineUp (  =5.0, =15.0) BatchUp (  =2.0, =15.0) Q3Q Q 3 +H Q +H C Improve 72.4%32.6%93.8%39.4%67.7%20.2%92.4%39.4% Q4Q Q 4 +H Q +H C Improve 66.2%15.5%78.2%19.9%47.8%6.9%77.2%16.4% Short-term context helps system improve retrieval accuracy BayesInt better than FixInt; BatchUp better than OnlineUp

31 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Using Clickthrough Data Only Q3Q Q 3 +H C Improve81.9%37.1% Q4Q Q 4 +H C Improve72.6%18.1% BayesInt (  =0.0, =5.0) Clickthrough is the major contributor 13.9% 67.2%Improve Q 4 +H C Q4Q4 42.4%99.7%Improve Q 3 +H C Q3Q3 Performance on unseen docs -4.1%15.7%Improve Q 4 +H C Q4Q4 23.0%23.8%Improve Q 3 +H C Q3Q3 Snippets for non-relevant docs are still useful!

32 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Sensitivity of BatchUp Parameters BatchUp is stable with different parameter settings Best performance is achieved when  =2.0; =15.0

33 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, A User Study of Implicit Feedback UCAIR toolbar (a client-side personalized search agent using implicit feedback) is used in this study 6 participants use UCAIR toolbar to do web search 32 topics are selected from TREC Web track and Terabyte track Participants evaluate explicitly the relevance of top 30 search results from Google and UCAIR

34 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, UCAIR Outperforms Google: Precision at N Docs Ranking Method Google UCAIR Improvement8.0%17.8%20.2%21.8% More user interactions  better user models  better retrieval accuracy

35 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, UCAIR Outperforms Google: PR Curve

36 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Scenario 2: Use the Entire History of a User [Tan et al. 06] Challenge: Search log is noisy –How do we handle the noise? –Can we still improve performance? Solution: –Assign weights to the history data (Cosine, EM algorithm) Conclusions: –All the history information is potentially useful –Most helpful for recurring queries –History weighting is crucial (EM better than Cosine)

37 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Algorithm Illustration

38 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Sample Results: EM vs. Baseline History is helpful and weighting is important

39 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Sample Results: Different Weighting Methods EM is better than Cosine; hybrid is feasible

40 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What You Should Know All search history information helps Clickthrough information is especially useful; it’s useful even when the actual document is non- relevant Recurring queries get more help, but fresh queries can also benefit from history information

41 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Explicit Feedback [Shen et al. 05, Tan et al. 07]

42 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Term Feedback for Information Retrieval with Language Models Bin Tan, Atulya Velivelli, Hui Fang, ChengXiang Zhai University of Illinois at Urbana-Champaign

43 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Problems with Doc-Based Feedback A relevant document may contain non-relevant parts None of the top-ranked documents is relevant User indirectly controls the learned query model

44 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What about Term Feedback? Present a list of terms to a user and asks for judgments  –More direct contribution to estimating  q –Works even when no relevant document on top Challenges: –How do we select terms to present to a user? –How do we exploit term feedback to improve our estimate of  q ?

45 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Improve  q with Term Feedback Query Retrieval Engine d d User Document collection Term Feedback Models Improved estimate of  q Term Judgments Term Extraction Terms

46 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Feedback Term Selection General (old) idea: –The original query is used for an initial retrieval run –Feedback terms are selected from top N documents New idea: –Model subtopics –Select terms to represent every subtopic well –Benefits Avoid bias in term feedback Infer relevant subtopics, thus achieve subtopic feedback

47 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, ored expl un User-Guided Query Model Refinement User Explored area Document space Inferred topic preference direction Most promising new topic areas to move to T1 T2 T3 t11 t12 t21 t22 t31 t32 …

48 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Collaborative Estimation of  q qq d1d2d3…dNd1d2d3…dN top N docs ranked by D(  q ||  d ) t1t2t3…tLt1t2t3…tL judged feedback terms C 1 :0.2 C 2 :0.1 C 3 :0.3 … C K :0.1 weighted clusters q rank docs by D(  q ’ ||  d ) q’q’ q refined query model t1t2t3…tLt1t2t3…tL feedback terms C1C2C3…CKC1C2C3…CK Subtopic clusters P(w|  1 ) P(w|  2 ) P(w|  k ) TFB P(t 1 |  TFB )=0.2 … P(t3|  TFB )=0.1 … Original  q P(w|  q ) CFB P(w|  CFB )=0.2*P(w|  1 )+ 0.1*P(w|  2 )+ … TCFB

49 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Discovering Subtopic Clusters with PLSA [Hofmann 99, Zhai et al. 04] Document d Theme  1 Theme  k Theme  2 … Background B traffic 0.3 railway Tunnel 0.1 fire 0.05 smoke tunnel 0.2 amtrack 0.1 train Is 0.05 the 0.04 a kk 11 22 B B 1 - B  d,1  d, k  d,2 W “Generating” word w in doc d in the collection Query = “transportation tunnel disaster” Maximum Likelihood Estimator (EM Algorithm)

50 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Selecting Representative Terms –Original query terms excluded –Shared terms assigned to most likely clusters Cluster 1Cluster 2Cluster 3 tunnel transport 1. traffic 2. railwai 3. harbor 4. rail 5. bridg 6. kilomet truck 7. construct …… tunnel 2. fire 3. truck 4. french 5. smoke 6. car 7. italian 8. firefight 9. blaze 10. blanc …… tunnel 1. transport 2. toll 3. amtrak 4. train 5. airport 6. turnpik 7. lui 8. jersei 9. pass …… L

51 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, User Interface for Term Feedback Cluster 1Cluster 3Cluster 2Cluster 1Cluster 2Cluster 3Cluster 1Cluster 2

52 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Experiment Setup TREC 2005 HARD Track AQUAINT corpus (3GB) 50 hard query topics NIST assessors spend up to 3 min on each topic providing feedback using Clarification Form (CF) Submitted CFs: 1x48, 3x16, 6x8 Baseline: KL-divergence retrieval method with 5 pseudo- feedback docs 48 terms generated from top 60 docs of baseline

53 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Retrieval Accuracy Comparison 1C: 1x48 3C: 3x16 6C: 6x8 (except for CFB1C) Baseline < TFB < CFB < TCFB CFB1C: user feedback plays no role Base -line TFBCFBTCFB 1C3C6C1C3C6C1C3C6C MAP RR MAP% 0%31.5% 26.9%16.0%39.3%37.4%25.1%41.1%38.8%

54 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Reduction of # Terms Presented TFBCFBTCFB #terms1C3C6C3C6C3C6C #terms=12: 1x12/3x4/6x2

55 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Clarification Form Completion Time More than half completed in just 1 min

56 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Term Relevance Judgment Quality CF Type1x483x166x8 #checked terms #rel. terms #rel. checked terms precision recall [Zaragoza et al. 04] Term relevance

57 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Had the User Checked all “ Relevant Terms ”… TFB > TFB > TFB > CFB > CFB > TCFB > TCFB > 0.341

58 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Comparison to Relevance Feedback # FB TCFB3C MAP equivalence: TCFB3C = Rel FB with 5 docs

59 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Term Feedback Help Difficult Topics No rel docs In top 5

60 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Related Work Early work: [Harman 88], [Spink 94], [Koenemann & Belkin 96]… More recent: [Ruthven03], [Anick03], … Main differences: –Language model –Consistently effective

61 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Conclusions and Future Work A novel way of improving query model estimation through term feedback –active feedback based on subtopics –user-system collaboration –achieves large performance improvement over non-feedback baseline with small amount of user effort –can compete with relevance feedback, esp. in a situation when the latter is unable to help To explore more complex interaction processes – Combination of term feedback and relevance feedback – Incremental feedback

62 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What You Should Know Term feedback can be quite useful when the query is difficult and relevance feedback isn’t feasible Language models can address weighting well in term feedback

63 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Active Feedback in Ad Hoc IR Xuehua Shen, ChengXiang Zhai Department of Computer Science University of Illinois, Urbana-Champaign

64 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Normal Relevance Feedback (RF) Feedback Judgments: d 1 + d 2 - … d k - Query Retrieval System Top K Results d d … d k 0.5 User Document Collection

65 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Document Selection in RF Feedback Judgments: d 1 + d 2 - … d k - Query Retrieval System Which k docs to present ? User Document Collection Can we do better than just presenting top-K? (Consider diversity…)

66 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Active Feedback (AF) An IR system actively selects documents for obtaining relevance judgments If a user is willing to judge K documents, which K documents should we present in order to maximize learning effectiveness?

67 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Outline Framework and specific methods Experiment design and results Summary and future work

68 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, A Framework for Active Feedback Consider active feedback as a decision problem –Decide K documents (D) for relevance judgment Formalize it as an optimization problem –Optimize the expected learning benefits (loss) by requesting relevance judgments on D from the user Consider two cases of loss function according to the interaction between documents –Independent loss: value of each judged document for learning is independent on each other –Dependent loss

69 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Independent Loss Rank docs according to expected loss of each individual doc and then select top K docs Top K Constant loss for any relevant and non-relevant docs Smaller loss for relevant docs A doc is more useful for learning if the prediction of relevance is more uncertain Uncertainty Sampling

70 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Dependent Loss Heuristics: consider relevance first, then diversity First select Top N docs of baseline retrieval Cluster N docs into K clusters K Cluster Centroid MMR … Model diversity and relevance Gapped Top K Pick one doc every G+1 docs

71 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Illustration of Three AF Methods Top-K (normal feedback) … Gapped Top-K K-Cluster Centroid Aiming at high diversity …

72 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Evaluating Active Feedback Query Select K Docs K docs Judgment File + Judged Docs Initial Results No Feedback (Top-k, Gapped, Clustering) Feedback Results

73 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Retrieval Methods (Lemur toolkit) Query Q Document D Results KL Divergence Feedback Docs F={d 1, …, d n } Active Feedback Default parameter settings unless otherwise stated Mixture Model Feedback Only learn from relevant docs

74 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Comparison of Three AF Methods CollectionActive FB Method #AFRe l per topic Include judged docs HARD Baseline/ Pseudo FB/ Top-K Gapped Clustering AP88-89 Baseline/ Pseudo FB/ Top-K Gapped Clustering Top-K is the worst! Clustering uses fewest relevant docs

75 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Appropriate Evaluation of Active Feedback New DB (AP88-89, AP90) Original DB with judged docs (AP88-89, HARD) Original DB without judged docs Can’t tell if the ranking of un-judged documents is improved Different methods have different test documents See the learning effect more explicitly But the docs must be similar to original docs

76 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Retrieval Performance on AP90 Dataset Metho d Baselin e Pseudo FB Top KGapped Top K K Cluster Centroid MAP Top-K is consistently the worst!

77 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Feedback Model Parameter  Factor  parameter can amplify the effect of feedback

78 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Summary Introduce the active feedback problem Propose a preliminary framework and three methods (Top-k, Gapped Top-k, Clustering) Study the evaluation strategy Experiment results show that –Presenting the top-k is not the best strategy –Clustering can generate fewer, higher quality feedback examples

79 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Future Work Explore other methods for active feedback Develop a general framework Combine pseudo feedback and active feedback

80 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What You Should Know What is active feedback Top-k isn’t a good strategy for active feedback; diversifying the results is beneficial

81 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Learn from Web Search Logs to Organize Search Results Xuanhui Wang and ChengXiang Zhai Department of Computer Science University of Illinois, Urbana-Champaign

82 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Motivation Search engine utility = Ranking accuracy + Result presentation + … Lots of research on improving ranking accuracy Relatively little work on improving result presentation What’s the best way to present search results?

83 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Ranked List Presentation

84 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, However, when the query is ambiguous… Query = Jaguar Car Software Animal Unlikely optimal for any particular user!

85 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Cluster Presentation (e.g., [Hearst et al. 96, Zamir & Etzioni 99]) From

86 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Deficiencies of Data-Driven Clustering Different users may prefer different ways to group the results. E.g., query=“area codes” –“phone codes” vs “zip codes” –“international codes” vs “local codes” Cluster labels may not be informative to help a user choose the right cluster. E.g., label = “panthera onca” Need to group search results from a user’s perspective

87 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Our Idea: User-Oriented Clustering User-oriented clustering: –Partition search results according to the aspects interesting to users –Label each aspect with words meaningful to users Exploit search logs to do both –Partitioning Learn “interesting aspects” of an arbitrary query Classify results into these aspects –Labeling Learn “representative queries” of the identified aspects Use representative queries to label the aspects

88 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Rest of the Talk General Approach Technical Details Experiment Results

89 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Illustration of the General Idea query =“ car” car rental car pricing used car hertz car rental car accidents car audio car crash … Retrieval (over log) 1.{car rental, hertz car rental…} 2.{car pricing, used car,…} 3.{car accidents, car crash, …} 4.{car audio, car stereo, …} 5.… Clustering … Results 1.{car rental, hertz car rental,…} … 2.{car pricing, used car,…} 3.{car accidents, car crash,…} … Categorization Car rental Used cars Car accidents

90 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Query Search History Collection Query pseudo doc1 Query pseudo doc2 Clustering Query Aspect 1 Query Aspect k … Results … Retrieval Similar Queries Categorization … Labeling … Label 1 Label 2 User-Oriented Clustering via Log Mining

91 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Query Search History Collection Query pseudo doc1 Query pseudo doc2 … Results … Implementation Strategy Retrieval Similar Queries BM25 Lemur query+clicked snippets Pooling identical queries Clustering Query Aspect 1 Query Aspect k Star clustering Labeling … Label 1 Label 2 Center query Categorization … Centroid-based

92 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, More Details: Search Engine Log Record user activities (queries, clicks) Reflect user information needs Valuable resources for learning to improve search engine utility sessions

93 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Recover snippets More Details: Build History Collection Pooling “car rental” … “jaguar car”… … session n For every query (e.g., car rental) “car rental”“Car rental, rental cars” …,“National car rental” …, … “jaguar car”“jaguar, car, parts”…, … …… History Collection Clicked urls U1, U2, …

94 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, More Details: Star Clustering [Aslam et al. 04] Form a similarity graph -TF-IDF weight vectors -Cosine similarity -Thresholding 2. Iteratively identify a “star center” and its “satellites” “Star center” query serves as a label for a cluster

95 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Centroid-Based Classifier Represent each query doc as a term vector (TF-IDF weighting) Compute a centroid vector for each cluster/aspect Assign a new result vector to the cluster whose centroid is the closest to the new vector Aspect 1 Aspect 2 Aspect 3

96 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Evaluation: Data Preparation Log data: May 2006 search log released by Microsoft Live Labs First 2/3 to simulate history; last 1/3 to simulate future queries History collection (169,057 queries;3.5 clicked URLs/query) “Future” collection is further split into two sets for validation and testing Test case: a session with more than 4 clicks and at least 100 matching queries in history (172 and 177 test cases in two test sets) Use clicked URLs to approximate relevant documents [Joachims, 2002]

97 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Experiment Design Baseline method –the original search engine ranking Cluster-based method –Traditional method solely based on content Log-based method –Our method based on search logs Evaluation –Based on a user’s perceived ranking accuracy –A user is assumed to first view the cluster with largest number of relevant docs –Measures documents Mean Reciprocal Rank (MRR) of the first relevant document

98 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Overall Comparison Log-based >> baseline Log-based >> cluster-based

99 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Diversity Analysis Do queries with diverse results benefit more? Bin by size ratios of the two largest clusters Queries with diverse results benefit more Primary/Secondary cluster size ratio more diverse

100 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Query Difficulty Analysis Do difficult queries benefit more? Bin by Mean Average Precisions (MAPs) Difficult queries benefit more more difficult

101 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Effectiveness of Learning more history information

102 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Sample Results: Partitioning Log-based method and regular clustering partition the results differently Query: “area codes” “International codes” or “local codes” “Phone codes” or “zip codes”

103 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Sample Results: Labeling Query: apple Query: jaguar

104 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Related Work Categorization-based (e.g.,Chen & Dumais 00]) –Labels are meaningful to users –Partitioning may not match a user’s perspective Faceted search and browsing (e.g.,[Yee et al. 03]) –Labels are meaningful to users –Partitioning is generally useful for a user –Need faceted metadata Rather than pre-specify fixed categories/metadata, we learn them dynamically from search log

105 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Conclusions and Future Work Proposed a general strategy for organizing search results based on interesting topic aspects learned from search log Experimented with a way to implement the strategy Results show that –User-oriented clustering is better than data-oriented clustering –Particularly help difficult topics and topics with diverse results Future directions –Mixture of data-driven and user-driven clustering –Study user interaction/feedback with cluster interface –Use general search log to “smooth” personal search log –Query-sensitive result presentation

106 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, What You Should Know Search history for multiple users can be combined to benefit a particular user’s search Difference between user-oriented result organization and data-oriented organization and their advantages and disadvantages How to evaluate clustering results indirectly based on the perceived precision

107 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Future Research Directions in Personalized Search Robust personalization: –Optimization framework for progressive personalization (gradually become more and more aggressive in using context/history information) More in-depth analysis of implicit feedback information –Why does a user add a query term and then drop it after viewing a particular document? More computer-user dialogue to help bridging the vocabulary gap Generally, aim at helping improve performance for difficult topics What’s the right architecture for supporting personalized search?

108 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Roadmap This lecture: Personalized search (understanding users) Next lecture: NLP for IR (understanding documents)

109 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, User-Centered Search Engine “java” Personalized search agent WEB Search Engine Search Engine Desktop Files Personalized search agent “java”... Viewed Web pages Query History A search agent can know about a particular user very well

110 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, User-Centered Adaptive IR (UCAIR) A novel retrieval strategy emphasizing – user modeling (“user-centered”) – search context modeling (“adaptive”) – interactive retrieval Implemented as a personalized search agent that –sits on the client-side (owned by the user) –integrates information around a user (1 user vs. N sources as opposed to 1 source vs. N users) –collaborates with each other –goes beyond search toward task support

111 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, A Simple UCAIR System (UCAIR Toolbar) Search Engine Search History Log (e.g.,past queries, clicked results) Query Modification Result Re-Ranking User Modeling Result Buffer UCAIR User query results clickthrough…

112 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, Challenges in UCAIR What’s an appropriate retrieval framework for UCAIR? How do we optimize retrieval performance in interactive retrieval? How do we develop robust and accurate retrieval models to exploit user information and search context? How do we evaluate UCAIR methods? ……

113 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, The Rest of the Talk Part I: A risk minimization framework for UCAIR Part II: Improve document ranking with implicit feedback Part III: User-specific summarization of search results Joint work with Xuehua Shen, Bin Tan, and Qiaozhu Mei


Download ppt "2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 1 龙星计划课程 : 信息检索 Personalized Search & User Modeling ChengXiang Zhai."

Similar presentations


Ads by Google