Evaluation of Relevance Feedback Algorithms for XML Retrieval Silvana Solomon 27 February 2007 Supervisor: Dr. Ralf Schenkel.

Slides:

Advertisements

Similar presentations

Even More TopX: Relevance Feedback Ralf Schenkel Joint work with Osama Samodi, Martin Theobald.

Advertisements

Information Extraction Lecture 4 – Named Entity Recognition II CIS, LMU München Winter Semester Dr. Alexander Fraser, CIS.

Retrieval Evaluation J. H. Wang Mar. 18, Outline Chap. 3, Retrieval Evaluation –Retrieval Performance Evaluation –Reference Collections.

1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.

Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.

Exercising these ideas  You have a description of each item in a small collection. (30 web sites)  Assume we are looking for information about boxers,

Section Based Relevance Feedback Student: Nat Young Supervisor: Prof. Mark Sanderson.

Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.

GENERATING AUTOMATIC SEMANTIC ANNOTATIONS FOR RESEARCH DATASETS AYUSH SINGHAL AND JAIDEEP SRIVASTAVA CS DEPT., UNIVERSITY OF MINNESOTA, MN, USA.

1 Entity Ranking Using Wikipedia as a Pivot (CIKM 10’) Rianne Kaptein, Pavel Serdyukov, Arjen de Vries, Jaap Kamps 2010/12/14 Yu-wen,Hsu.

Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)

DYNAMIC ELEMENT RETRIEVAL IN A STRUCTURED ENVIRONMENT MAYURI UMRANIKAR.

1 Configurable Indexing and Ranking for XML Information Retrieval Shaorong Liu, Qinghua Zou and Wesley W. Chu UCLA Computer Science Department {sliu, zou,

Re-ranking Documents Segments To Improve Access To Relevant Content in Information Retrieval Gary Madden Applied Computational Linguistics Dublin City.

Routing of Structured Queries in Large-Scale Distributed Systems Workshop on Large-Scale Distributed Systems for Information Retrieval ACM.

Retrieval Evaluation: Precision and Recall. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity.

© Anselm Spoerri Lecture 13 Housekeeping –Term Projects Evaluations –Morse, E., Lewis, M., and Olsen, K. (2002) Testing Visual Information Retrieval Methodologies.

SIMS 202 Information Organization and Retrieval Prof. Marti Hearst and Prof. Ray Larson UC Berkeley SIMS Tues/Thurs 9:30-11:00am Fall 2000.

Hybrid XML Retrieval Revisited Jovan Pehcevski PhD Candidate School of CS and IT, RMIT University

Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Affinity Rank Yi Liu, Benyu Zhang, Zheng Chen MSRA.

Web Search – Summer Term 2006 II. Information Retrieval (Basics Cont.) (c) Wolfgang Hürst, Albert-Ludwigs-University.

The Relevance Model  A distribution over terms, given information need I, (Lavrenko and Croft 2001). For term r, P(I) can be dropped w/o affecting the.

Minimal Test Collections for Retrieval Evaluation B. Carterette, J. Allan, R. Sitaraman University of Massachusetts Amherst SIGIR2006.

INEX – a broadly accepted data set for XML database processing? Pavel Loupal, Michal Valenta.

1 Retrieval and Feedback Models for Blog Feed Search SIGIR 2008 Advisor ： Dr. Koh Jia-Ling Speaker ： Chou-Bin Fan Date ：

1 Formal Models for Expert Finding on DBLP Bibliography Data Presented by: Hongbo Deng Co-worked with: Irwin King and Michael R. Lyu Department of Computer.

Redeeming Relevance for Subject Search in Citation Indexes Shannon Bradshaw The University of Iowa

A Simple Unsupervised Query Categorizer for Web Search Engines Prashant Ullegaddi and Vasudeva Varma Search and Information Extraction Lab Language Technologies.

A Comparison of Statistical Significance Tests for Information Retrieval Evaluation CIKM´07, November 2007.

Modern Information Retrieval: A Brief Overview By Amit Singhal Ranjan Dash.

Topical Crawlers for Building Digital Library Collections Presenter: Qiaozhu Mei.

Querying Structured Text in an XML Database By Xuemei Luo.

TopX 2.0 at the INEX 2009 Ad-hoc and Efficiency tracks Martin Theobald Max Planck Institute Informatics Ralf Schenkel Saarland University Ablimit Aji Emory.

April 14, 2003Hang Cui, Ji-Rong Wen and Tat- Seng Chua 1 Hierarchical Indexing and Flexible Element Retrieval for Structured Document Hang Cui School of.

1 Information Retrieval Acknowledgements: Dr Mounia Lalmas (QMW) Dr Joemon Jose (Glasgow)

Probabilistic Query Expansion Using Query Logs Hang Cui Tianjin University, China Ji-Rong Wen Microsoft Research Asia, China Jian-Yun Nie University of.

Controlling Overlap in Content-Oriented XML Retrieval Charles L. A. Clarke School of Computer Science University of Waterloo Waterloo, Canada.

JJE: INEX XML Competition Bryan Clevenger James Reed Jon McElroy.

1 Relevance Ranking in the Scholarly Domain Dr. Tamar Sadeh LIBER Conference Tartu, Estonia, June 2012 Dr. Tamar Sadeh LIBER Conference Tartu, Estonia,

Web Image Retrieval Re-Ranking with Relevance Model Wei-Hao Lin, Rong Jin, Alexander Hauptmann Language Technologies Institute School of Computer Science.

Search Engine Architecture

BNCOD07Indexing & Searching XML Documents based on Content and Structure Synopses1 Indexing and Searching XML Documents based on Content and Structure.

LANGUAGE MODELS FOR RELEVANCE FEEDBACK Lee Won Hee.

Users and Assessors in the Context of INEX: Are Relevance Dimensions Relevant? Jovan Pehcevski, James A. Thom School of CS and IT, RMIT University, Australia.

1 FollowMyLink Individual APT Presentation Third Talk February 2006.

1 Using The Past To Score The Present: Extending Term Weighting Models with Revision History Analysis CIKM’10 Advisor ： Jia Ling, Koh Speaker ： SHENG HONG,

21/11/20151Gianluca Demartini Ranking Clusters for Web Search Gianluca Demartini Paul–Alexandru Chirita Ingo Brunkhorst Wolfgang Nejdl L3S Info Lunch Hannover,

Chapter 8 Evaluating Search Engine. Evaluation n Evaluation is key to building effective and efficient search engines  Measurement usually carried out.

Measuring How Good Your Search Engine Is. *. Information System Evaluation l Before 1993 evaluations were done using a few small, well-known corpora of.

C.Watterscs64031 Evaluation Measures. C.Watterscs64032 Evaluation? Effectiveness? For whom? For what? Efficiency? Time? Computational Cost? Cost of missed.

Advantages of Query Biased Summaries in Information Retrieval by A. Tombros and M. Sanderson Presenters: Omer Erdil Albayrak Bilge Koroglu.

1 Evaluating High Accuracy Retrieval Techniques Chirag Shah,W. Bruce Croft Center for Intelligent Information Retrieval Department of Computer Science.

26/01/20161Gianluca Demartini Ranking Categories for Faceted Search Gianluca Demartini L3S Research Seminars Hannover, 09 June 2006.

The Loquacious ( 愛說話 ) User: A Document-Independent Source of Terms for Query Expansion Diane Kelly et al. University of North Carolina at Chapel Hill.

TO Each His Own: Personalized Content Selection Based on Text Comprehensibility Date: 2013/01/24 Author: Chenhao Tan, Evgeniy Gabrilovich, Bo Pang Source:

CS798: Information Retrieval Charlie Clarke Information retrieval is concerned with representing, searching, and manipulating.

Usefulness of Quality Click- through Data for Training Craig Macdonald, ladh Ounis Department of Computing Science University of Glasgow, Scotland, UK.

CS791 - Technologies of Google Spring A Webbased Kernel Function for Measuring the Similarity of Short Text Snippets By Mehran Sahami, Timothy.

Enterprise Track: Thread-based Retrieval Enterprise Track: Thread-based Retrieval Yejun Wu and Douglas W. Oard Goal Explore -- document expansion.

University Of Seoul Ubiquitous Sensor Network Lab Query Dependent Pseudo-Relevance Feedback based on Wikipedia 전자전기컴퓨터공학 부 USN 연구실 G

Research Paper Sections

Developments in Evaluation of Search Engines

Search Engine Architecture

CADIAL search engine at INEX

Toshiyuki Shimizu (Kyoto University)

IR Theory: Evaluation Methods

INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID

Search Engine Architecture

Relevance and Reinforcement in Interactive Browsing

Presentation transcript:

Evaluation of Relevance Feedback Algorithms for XML Retrieval Silvana Solomon 27 February 2007 Supervisor: Dr. Ralf Schenkel

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Outline Short introduction Motivation & Goals Evaluating retrieval effectiveness INEX tool Evaluation methodology Results

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Introduction Path to the result sec „ The IR process is composed …“ article body sec subsec „ For small collections …“ frontmatter sec subsec pp p „ Figure 1 outlines …“ author „ Ian Ruthven “ Content of result citation „ D. Harman “ backmatter (3) feedback (4) expanded query Feedback XML Search Engine (1) query (2) results (5) results of expanded query

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Motivation Best way to compare feedback algorithms? Cannot use standard evaluation tools on feedback results Goals:  Analyze evaluation methods  Develop an evaluation tool

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluating Retrieval Effectiveness Document collection Topics set Assessments set Human assessors Metrics INEX: INitiative for the Evaluation of XML Retrieval 2006 document collection: 600,000 Wikipedia documents

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 INEX Tool: EvalJ Tool for evaluation of information retrieval experiments Implements a set of metrics used for evaluation Limitations: cannot measure improvement of runs produced with feedback

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 RF Evaluation – Ranking Effect Baseline run doc[1]/bdy[1] doc[2]/bdy[1] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[1] Mark in top results relevant doc[3] doc[8]/bdy[1]/article[3] doc[3] doc[8]/bdy[1]/article[3] doc[7]/article[3] push the known relevant results to the top of the element ranking artificially improves RP figures doc[2]/bdy[1]/article[1]

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 RF Evaluation – Feedback Effect measure improvement on unseen relevant elements not directly tested Modify FB run Evaluate untrained results Baseline run doc[1]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[3] doc[8]/bdy[1]/article[3] Mark in top results relevant

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (1) 1. Standard text IR: freezing known results at the top  independent results assumption 2. New approach: remove known results+X from the collection  resColl-result : remove results only (~doc retrieval)  resColl-desc : remove results+descendants  resColl-anc : remove results+ancestors  resColl-path : remove results+desc+anc  resColl-doc : remove whole doc with known results

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Freezing: Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4]

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[3] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[4]/bdy[1]/ article[4] resColl-path:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[4]/bdy[1]/ article[4] resColl-path:

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Best Evaluation Methodology? sec „ The IR process is composed …“ article body sec subsec „ For small collections …“ frontmatterbackmatter sec subsec p p P „ Figure 1 outlines …“ author „ Ian Ruthven “ citation „ D. Harman “ resColl-path

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Testing Evaluated Results Standard method: average – problems: Topic-id Avg. Baseline Modified feedback t-test & Wilcoxon signed-rank test: gives probability p that the baseline run is better than the feedback run experiment significant if p<0.05 or p<0.01

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Results (1) Evaluation mode: resColl-path Feedback fileINEX metric Abs. improv. Rel. improv. T-testWSR TopX_CO_Content.xml xfirm_r1_cosc3s.xml xfirm_r1_cosc5.xml xfirm_r1_cosc3.xml xfirm_r1_coc3s3.xml xfirm2_r2_cop4.xml xfirm2_r2_cot40.xml xfirm2_r2_cot10.xml xfirm_r1_coc3.xml xfirm_r1_coc10.xml

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Results (2) Comparison of evaluation techniques based on relative improvement w.r.t. baseline run freezingresColl- anc resColl- desc resColl- doc resColl- path resColl- res c3s TopX c3s TopXc5c3s c5 TopXc5 TopX c3 TopX = TopX_CO_Content.xml c3 = xfirm_r1_cosc3.xml c3s = xfirm_r1_cosc3s.xml c5 = xfirm_r1_cosc5.xml

Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Conclusions & Future Work Evaluation based on different techniques & metrics Correct improvement measurement Not solved: comparing several systems with different output Maybe a hybrid evaluation mode