1 Evaluating High Accuracy Retrieval Techniques Chirag Shah,W. Bruce Croft Center for Intelligent Information Retrieval Department of Computer Science.

Slides:



Advertisements
Similar presentations
Term Level Search Result Diversification DATE : 2013/09/11 SOURCE : SIGIR’13 AUTHORS : VAN DANG, W. BRUCE CROFT ADVISOR : DR.JIA-LING, KOH SPEAKER : SHUN-CHEN,
Advertisements

1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Improved TF-IDF Ranker
Query Dependent Pseudo-Relevance Feedback based on Wikipedia SIGIR ‘09 Advisor: Dr. Koh Jia-Ling Speaker: Lin, Yi-Jhen Date: 2010/01/24 1.
Evaluating Search Engine
How to Make Manual Conjunctive Normal Form Queries Work in Patent Search Le Zhao and Jamie Callan Language Technologies Institute School of Computer Science.
Search Engines and Information Retrieval
Evaluating Evaluation Measure Stability Authors: Chris Buckley, Ellen M. Voorhees Presenters: Burcu Dal, Esra Akbaş.
Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.
Using Information Extraction for Question Answering Done by Rani Qumsiyeh.
Reference Collections: Task Characteristics. TREC Collection Text REtrieval Conference (TREC) –sponsored by NIST and DARPA (1992-?) Comparing approaches.
Retrieval Evaluation: Precision and Recall. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity.
Queensland University of Technology An Ontology-based Mining Approach for User Search Intent Discovery Yan Shen, Yuefeng Li, Yue Xu, Renato Iannella, Abdulmohsen.
Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.
An investigation of query expansion terms Gheorghe Muresan Rutgers University, School of Communication, Information and Library Science 4 Huntington St.,
The Relevance Model  A distribution over terms, given information need I, (Lavrenko and Croft 2001). For term r, P(I) can be dropped w/o affecting the.
Mining and Summarizing Customer Reviews
Multi-Style Language Model for Web Scale Information Retrieval Kuansan Wang, Xiaolong Li and Jianfeng Gao SIGIR 2010 Min-Hsuan Lai Department of Computer.
Evaluating the Contribution of EuroWordNet and Word Sense Disambiguation to Cross-Language Information Retrieval Paul Clough 1 and Mark Stevenson 2 Department.
Title Extraction from Bodies of HTML Documents and its Application to Web Page Retrieval Microsoft Research Asia Yunhua Hu, Guomao Xin, Ruihua Song, Guoping.
Probabilistic Model for Definitional Question Answering Kyoung-Soo Han, Young-In Song, and Hae-Chang Rim Korea University SIGIR 2006.
A Comparative Study of Search Result Diversification Methods Wei Zheng and Hui Fang University of Delaware, Newark DE 19716, USA
Finding Similar Questions in Large Question and Answer Archives Jiwoon Jeon, W. Bruce Croft and Joon Ho Lee Retrieval Models for Question and Answer Archives.
1 Formal Models for Expert Finding on DBLP Bibliography Data Presented by: Hongbo Deng Co-worked with: Irwin King and Michael R. Lyu Department of Computer.
Linking Wikipedia to the Web Antonio Flores Bernal Department of Computer Sciencies San Pablo Catholic University 2010.
A Simple Unsupervised Query Categorizer for Web Search Engines Prashant Ullegaddi and Vasudeva Varma Search and Information Extraction Lab Language Technologies.
A Probabilistic Graphical Model for Joint Answer Ranking in Question Answering Jeongwoo Ko, Luo Si, Eric Nyberg (SIGIR ’ 07) Speaker: Cho, Chin Wei Advisor:
1 Query Operations Relevance Feedback & Query Expansion.
MIRACLE Multilingual Information RetrievAl for the CLEF campaign DAEDALUS – Data, Decisions and Language, S.A. Universidad Carlos III de.
Retrieval Models for Question and Answer Archives Xiaobing Xue, Jiwoon Jeon, W. Bruce Croft Computer Science Department University of Massachusetts, Google,
21/11/2002 The Integration of Lexical Knowledge and External Resources for QA Hui YANG, Tat-Seng Chua Pris, School of Computing.
Multilingual Relevant Sentence Detection Using Reference Corpus Ming-Hung Hsu, Ming-Feng Tsai, Hsin-Hsi Chen Department of CSIE National Taiwan University.
Efficiently Computed Lexical Chains As an Intermediate Representation for Automatic Text Summarization H.G. Silber and K.F. McCoy University of Delaware.
Predicting Question Quality Bruce Croft and Stephen Cronen-Townsend University of Massachusetts Amherst.
Web Image Retrieval Re-Ranking with Relevance Model Wei-Hao Lin, Rong Jin, Alexander Hauptmann Language Technologies Institute School of Computer Science.
Music Information Retrieval Information Universe Seongmin Lim Dept. of Industrial Engineering Seoul National University.
WIRED Week 3 Syllabus Update (next week) Readings Overview - Quick Review of Last Week’s IR Models (if time) - Evaluating IR Systems - Understanding Queries.
Automatic Identification of Pro and Con Reasons in Online Reviews Soo-Min Kim and Eduard Hovy USC Information Sciences Institute Proceedings of the COLING/ACL.
Chapter 8 Evaluating Search Engine. Evaluation n Evaluation is key to building effective and efficient search engines  Measurement usually carried out.
© 2004 Chris Staff CSAW’04 University of Malta of 15 Expanding Query Terms in Context Chris Staff and Robert Muscat Department of.
Semantic v.s. Positions: Utilizing Balanced Proximity in Language Model Smoothing for Information Retrieval Rui Yan†, ♮, Han Jiang†, ♮, Mirella Lapata‡,
Department of Software and Computing Systems Research Group of Language Processing and Information Systems The DLSIUAES Team’s Participation in the TAC.
Using a Named Entity Tagger to Generalise Surface Matching Text Patterns for Question Answering Mark A. Greenwood and Robert Gaizauskas Natural Language.
August 17, 2005Question Answering Passage Retrieval Using Dependency Parsing 1/28 Question Answering Passage Retrieval Using Dependency Parsing Hang Cui.
Information Retrieval using Word Senses: Root Sense Tagging Approach Sang-Bum Kim, Hee-Cheol Seo and Hae-Chang Rim Natural Language Processing Lab., Department.
Relevance-Based Language Models Victor Lavrenko and W.Bruce Croft Department of Computer Science University of Massachusetts, Amherst, MA SIGIR 2001.
Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq
Mining Dependency Relations for Query Expansion in Passage Retrieval Renxu Sun, Chai-Huat Ong, Tat-Seng Chua National University of Singapore SIGIR2006.
NTNU Speech Lab Dirichlet Mixtures for Query Estimation in Information Retrieval Mark D. Smucker, David Kulp, James Allan Center for Intelligent Information.
Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.
Learning to Estimate Query Difficulty Including Applications to Missing Content Detection and Distributed Information Retrieval Elad Yom-Tov, Shai Fine,
Topic by Topic Performance of Information Retrieval Systems Walter Liggett National Institute of Standards and Technology TREC-7 (1999)
Relevance Models and Answer Granularity for Question Answering W. Bruce Croft and James Allan CIIR University of Massachusetts, Amherst.
Divided Pretreatment to Targets and Intentions for Query Recommendation Reporter: Yangyang Kang /23.
The Loquacious ( 愛說話 ) User: A Document-Independent Source of Terms for Query Expansion Diane Kelly et al. University of North Carolina at Chapel Hill.
2/10/2016Semantic Similarity1 Semantic Similarity Methods in WordNet and Their Application to Information Retrieval on the Web Giannis Varelas Epimenidis.
Indri at TREC 2004: UMass Terabyte Track Overview Don Metzler University of Massachusetts, Amherst.
(Pseudo)-Relevance Feedback & Passage Retrieval Ling573 NLP Systems & Applications April 28, 2011.
A Framework to Predict the Quality of Answers with Non-Textual Features Jiwoon Jeon, W. Bruce Croft(University of Massachusetts-Amherst) Joon Ho Lee (Soongsil.
Usefulness of Quality Click- through Data for Training Craig Macdonald, ladh Ounis Department of Computing Science University of Glasgow, Scotland, UK.
Question Answering Passage Retrieval Using Dependency Relations (SIGIR 2005) (National University of Singapore) Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan,
PAIR project progress report Yi-Ting Chou Shui-Lung Chuang Xuanhui Wang.
Query Type Classification for Web Document Retrieval In-Ho Kang, GilChang Kim KAIST SIGIR 2003.
An Effective Statistical Approach to Blog Post Opinion Retrieval Ben He, Craig Macdonald, Jiyin He, Iadh Ounis (CIKM 2008)
Queensland University of Technology
Designing Cross-Language Information Retrieval System using various Techniques of Query Expansion and Indexing for Improved Performance  Hello everyone,
An Empirical Study of Learning to Rank for Entity Search
CS246: Information Retrieval
1Micheal T. Adenibuyan, 2Oluwatoyin A. Enikuomehin and 2Benjamin S
A Neural Passage Model for Ad-hoc Document Retrieval
Presentation transcript:

1 Evaluating High Accuracy Retrieval Techniques Chirag Shah,W. Bruce Croft Center for Intelligent Information Retrieval Department of Computer Science University of Massachusetts Presented By Yi-Ting Chen

2 Outline Introduction Analysis of the problem The approaches for modifying queries  Method1: giving more weight to the headwords.  Method2: Using clarity scores as weights.  Method3: Using clarity scores to find terms to expand with WordNet. Experiments and analysis Conclusion and future work

3 Introduction Ad-hoc retrieval and question answering: two major research streams in the present IR.  Different goals- Ad-hoc retrieval is mainly about retrieving a set of documents with good precision and recall, whereas QA focuses on getting one correct answer or a small set of answers with high accuracy. In this paper, tried to link these two achieving high accuracy with respect to the most relevant results.

4 Introduction HARD : TREC introduced a new track called High Accuracy Retrieval from Documents in This paper task also deals with the problem of getting high accuracy in retrieval, but with contrast to HARD, do not make use of any additional information. Studying how QA techniques can help in getting high accuracy for ad-hoc retrieval and propose a different measure for evaluation (MRR) instead of recall and precision.

5 Analysis of the problem To facilitate the evaluation of a system with high results in the rank list, QA system use mean reciprocal rank (MRR) as the measure. MRR id defined as the inverse of the rank of the retrieved result. Investigates the problem of achieving high accuracy with this perspective and analyzes the reasons behind low accuracy.

6 Analysis of the problem Baseline  TREC’s Tipster vol. I and II as datasets and topics (total 150) as queries (both title and description).  Used the language modeling framework for retrieval.  Used the Lemur toolkit for implementing our retrieval system.  The results of our baseline runs along with those of various QA systems of TREC-8 [26], TREC-9 [27], TREC- 10 [28], and TREC-11 [29] are given in table 1.

7

8 Analysis of the problem We can see that our baselines have a correct document in rank 1 about 40%. Look at the distributions of queries with respect to the rank. The average MRR could be increased by moving up relevant documents form lower ranks in the case of poorly performing queries, and by moving some of the big group of relevant document at rank 2 to rank 1.

9

10

11 Analysis of the problem Analyzing some bad queries :  We could improve the MRR value in all the cases by improving the query.  There were three problems that we could identify : ambiguous words in the query. mixtures of words of different importance in the query. Query-document information mis-match.  Not all the words are equally important in a query.  Not all kinds of expansion can help.

12

13 The approaches for modifying queries Method1: giving more weight to the headwords  Find the headword in a given query and giving it more weight than other words of the query.  Parse the query for part-of speech (POS) tagging.  Find the first noun phrase using POS information.  Consider the last noun of this noun phrase as the headword.  Give this headword more weight.

14 The approaches for modifying queries Method2: Using clarity scores as weights  The reason for poor retrieval is often the use of ambiguous words in the query.  To implement this idea, He used Cornen- Townsend etal.’s technique of finding query clarity scores.  Computing the relative entropy between a query language model and the corresponding collection language model.

15 The approaches for modifying queries Method2: Using clarity scores as weights  Find query clarity scores based on the technique.  Construct weighted queries with clarity score of each word as its weight as we want to give more weight to words that have high clarity scores. Method3: Using clarity scores to find terms to expand with WordNet.  Query-dataset mismatch is another factor that affects the accuracy of the retrieval.  It is not useful to expand every word of the given query.

16 The approaches for modifying queries Method3: Using clarity scores to find terms to expand with WordNet.  Find query clarity scores.  Divide all the terms into the following tree categories : Terms with high clarity scores should not be touched. Terms with very low clarity scores are likely to be very ambiguous. These words are ignored. Expand the terms whose scores are between the two limits of clarity scores using wordNet synonyms.

17 Experiments and analysis More than 700,000 documents from Tipster collection and taking more than 2GB of disk space. Extracted from topics making total 150 queries. The experiments were conducted on both title queries as well as description queries.

18 Experiments and analysis Title queries :

19 Experiments and analysis

20 Experiments and analysis

21 Experiments and analysis

22 Experiments and analysis Description query

23 Experiments and analysis

24 Experiments and analysis

25 Experiments and analysis

26 Experiments and analysis We can see in the runs for title queries that we pushed more queries to rank1, we also got more queries at ranks higher than 100. This means that while trying to improve the queries, we also hurt some queries. Runs for description queries be significantly better.

27 Conclusion and future work Our focus in the presented work was to improve the MRR of the first relevant document, the proposed techniques also helped in improving overall precision in many case. As one of our next steps in this research, we carried out experiments with relevance models. We noticed that while bringing some queries up in the rank list, the model also drove some other down in the list.