1. Date: 2012/5/28 Source: Alexander Kotov. al(CIKM’11) Advisor: Jia-ling, Koh Speaker: Jiun Jia, Chiou Interactive Sense Feedback for Difficult Queries

2.  Introduction Query Ambiguity  Interactive Sense Feedback Sense Detection Sense Presentation  Experiments Upper-bound performance User study  Conclusion Outline

3. Ambiguity of query terms is a common cause of inaccurate retrieval results. Challenging task: Fully automatic sense identification and disambiguation Ambiguous queries contain one or several polysemous terms. cardinals Query Birds Sports Clergy ?? Query ambiguity is one of the main reasons for poor retrieval results - Difficult queries are often ambiguous

4. Solution Diversification Target sense is minority sense Relevance Feedback Top documents irrelevant doesn’t help users submit ambiguous queries spend some time and effort perusing search results,not realizing that the sense of a polysemous term that they had in mind is not the most common sense in the collection being searched.

5. Can search systems improve the results for difficult queries by naturally leveraging user interaction to resolve lexical ambiguity? Ideally, sense suggestions can be presented as clarification questions “Did you mean as ?” Users can leverage their intelligence and world knowledge to interpret the signals from the system and make the final decision. search systems can infer the collection-specific senses of query terms

6. Compare algorithms based on their upper bound retrieval performance and select the best performing one. Propose several methods for concise representation of the discovered senses. Evaluate the effectiveness of each method with the actual retrieval performance of user sense selections.

7. Step 1 construct a contextual term similarity matrix Step 2 construct a query term similarity graph Step 3 Label and present the senses to the users Step 4 Update the query Language Model using user feedback

8. Algorithm w1 w2 w3 S:sparse matrix S ij : the strength of semantic relatedness of the words wi and wj in a document collection C. Mutual Information Hyperspace Analog to Language X w and X v are binary variables indicating whether w or v are present or absent in a document.

9. p(X w = 1)c(X w = 1)/N p(X w = 0)1 − p(X w = 1) p(X v = 1)c(X v = 1)/N p(X v = 0)1 − p(X v = 1) p(X w = 1, X v = 1)c(X w = 1, X v = 1)/N p(X w = 1, X v = 0)[c(X w = 1) − c(X w = 1, X v = 1)]/N p(X w = 0, X v = 1)[c(X v = 1) − c(X w = 1, X v = 1)]/N p(X w = 0, X v = 0)1− p(X w = 1, X v = 0) −p(X w = 0, X v = 1) − p(X w = 1, X v = 1) Two words tend to occur in the same document,the more semantically related they are. Mutual information measures the strength of association between the two words and can be considered as a measure of their semantic relatedness. MI Sum=1

10. Document 1 Word1(D1,D3,D4,D6) Word2(D2,D3,D4,D6) Word3( D1, D3, D6 ) Document 2 Document 3 Document 4 Document 5 Document 6 Word4(D1,D3,D4,D6)

11. HAL(Hyperspace Analog to Language): Constructing the HAL space for an n-term vocabulary involves traversing a sliding window of width w over each word in the corpus, ignoring punctuation, sentence and paragraph boundaries. HAL space matrix (H) for the sentence “the effects of pollution on the population” Slide window size=10 The eff of poll on the pop Center 5 4 3 2 1 5 words before and after the center word W=10 eff of poll on the pop W=10 1 2 3 4 5 CenterHAL space matrix for the sentence “the effects of pollution on the population” Center 5

12.  Global co-occurrence matrix: first produced by merging the row and column corresponding to each term in the HAL space matrix. Each term t corresponds to a row in the global co-occurrence matrix H t = {(t 1, c 1 ),..., (t m, c m )}: Number of co-occurrences of the term t with all other terms in the vocabulary.

13. Normalized to obtain the contextual term similarity matrix for the collection: theeffofpollonpop T11/347/34 5/34 T27/2005/204/203/201/20 T37/235/230 4/232/23 T47/244/245/240 3/24 T57/233/234/235/2304/23 T65/151/152/153/154/150 contextual term similarity matrix w1 w2 w3 w4 w5 w6 theeffofpollonpop T1177775 T2705431 T3750542 T4745053 T5734504 T6512340

14. Step 1 construct a contextual term similarity matrix Step 2 construct a query term similarity graph Step 3 Label and present the senses to the users Step 4 Update the query Language Model using user feedback

15. 2. For each query term construct a query term similarity graph Sense Detection Algorithm: Cluster the term graph Cluster the term graph (cluster = sense ) 0.25 0.55 0.2 Clustering algorithms: Community clustering (CC) Clustering by committee (CBC)

16. Step 1 construct a contextual term similarity matrix Step 2 construct a query term similarity graph Step 3 Label and present the senses to the users Step 4 Update the query Language Model using user feedback

17. 3.Label and present the senses to the users ─ using the top k terms with the highest probability in the sense language model ─ selecting a small number of the most representative terms from the sense language model as a sense label Sense Presentation : 1 0.01 0.02 0.05 0.03 0.11 0.02 0.03 0.01 0.1 0.04 0.02 0.06 3 4 5 5 6 Label: 41 2

18. Step 1 construct a contextual term similarity matrix Step 2 construct a query term similarity graph Step 3 Label and present the senses to the users Step 4 Update the query Language Model using user feedback

19. 4.Update the query Language Model using user feedback w1 w2 w3 w4 w5 0.055 Select “Fiscal”

20.  KL-divergence retrieval model θ q :query language model θ Di : document language model each document D i in the document collection C = {D 1,..., D m }. A B C D Θq (2) (3) (2) (3) query A B C D E ΘD (2) (2) (1) (3) (2) document A: 0.2*log(0.2/0.2)=0 B : 0.3*log(0.3/0.2)=0.1755 C : 0.2*log(0.2/0.1)=0.2 D KL =0.3755 D: 0.3*log(0.3/0.3)=0 E : 0 increase KL value: increase

21.  Datasets: 3 TREC collections  Upper bound experiments: try all detected senses for all query terms and study the potential of using sense feedback for improving retrieval results.  User study: present the labeled sense to the users and see whether users can recognize the best-performing sense; determine the retrieval performance of user selected senses.

22. ( for difficult topics ) * indicates statistically significant difference relative to KL (95% confidence level), according to the Wilcoxon signed-rank test. † indicates statistically significant difference relative to KL-PF (95% confidence level), according to the Wilcoxon signed-rank test.

23. Sense feedback improved more difficult queries than Pseudo feedback(PF) in all datasets TotalDiffNorm PFSF Diff+Norm+Diff+Norm+ AP993464194437 ROBUST04249741753789 AQUAINT501634426

24. Community Clustering (CC) outperforms Clustering by Committee (CBC) HAL scores are more effective than Mutual Information (MI) Sense Feedback performs better than PF on difficult query sets

25. Sample senses discovered by using the community clustering algorithm in combination with the HAL scores: cancer research different types of cancer cancer treatment cancer statistics in the US senses discovered for the term “cancer” in the query “radio waves and brain cancer”

26.  50 AQUAINT queries along with senses determined using CC and HAL  Senses presented as: 1, 2, 3 sense label terms using the labeling algorithm ( LAB1, LAB2, LAB 3 ) 3 and 10 terms with the highest score from the sense language model ( SLM3, SLM10 )  From all senses of all query terms users were asked to pick one sense using each of the sense presentation method p(term| )

27. Users selected the optimal query term for disambiguation for more than half of the queries Percentage of users selecting the optimal sense of the optimal term for sense feedback (in boldface) and the optimal term, but suboptimal sense (in parenthesis).

28. Users sense selections do not achieve the upper bound, but consistently improve over the baselines 0.2286 (0.2286)

29.  Interactive sense feedback as a new alternative feedback method  Proposed methods for sense detection and representation that are effective for both normal and difficult queries  Promising upper bound performance all collections  User studies demonstrated that users can recognize the best-performing sense in over 50% of the cases user-selected senses can effectively improve retrieval performance for difficult queries

30. END

31. Mean Average Precision (mAP) Supplementary material R q1 ={d 3,d 56, d 129 } Three relevant documents ranked at 3, 8, 15 Precisions are 1/3, 2/8, 3/15 Average Precision=(0.33+0.25+0.2)/3 =0.26 MAP=(0.26+0.58)/2=0.42 R q2 ={d 3, d 9, d 25,d 56,d 123 } Three relevant documents ranked at 3, 8, 15 Precisions are 1/1,2/3,3/6,4/10,5/15 Average Precision= (1+0.67+0.5+0.4+0.33)/5=0.58