Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation of Relevance Feedback Algorithms for XML Retrieval Silvana Solomon 27 February 2007 Supervisor: Dr. Ralf Schenkel.

Published byCalvin Snelling Modified over 9 years ago

Similar presentations

Presentation on theme: "Evaluation of Relevance Feedback Algorithms for XML Retrieval Silvana Solomon 27 February 2007 Supervisor: Dr. Ralf Schenkel."— Presentation transcript:

1 Evaluation of Relevance Feedback Algorithms for XML Retrieval Silvana Solomon 27 February 2007 Supervisor: Dr. Ralf Schenkel

2 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Outline Short introduction Motivation & Goals Evaluating retrieval effectiveness INEX tool Evaluation methodology Results

3 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Introduction Path to the result sec „ The IR process is composed …“ article body sec subsec „ For small collections …“ frontmatter sec subsec pp p „ Figure 1 outlines …“ author „ Ian Ruthven “ Content of result citation „ D. Harman “ backmatter (3) feedback (4) expanded query Feedback XML Search Engine (1) query (2) results (5) results of expanded query

4 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Motivation Best way to compare feedback algorithms? Cannot use standard evaluation tools on feedback results Goals:  Analyze evaluation methods  Develop an evaluation tool

5 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluating Retrieval Effectiveness Document collection Topics set Assessments set Human assessors Metrics INEX: INitiative for the Evaluation of XML Retrieval 2006 document collection: 600,000 Wikipedia documents

6 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 INEX Tool: EvalJ Tool for evaluation of information retrieval experiments Implements a set of metrics used for evaluation Limitations: cannot measure improvement of runs produced with feedback

7 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 RF Evaluation – Ranking Effect Baseline run doc[1]/bdy[1] doc[2]/bdy[1] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[1] Mark in top results relevant doc[3] doc[8]/bdy[1]/article[3] doc[3] doc[8]/bdy[1]/article[3] doc[7]/article[3] push the known relevant results to the top of the element ranking artificially improves RP figures doc[2]/bdy[1]/article[1]

8 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 RF Evaluation – Feedback Effect measure improvement on unseen relevant elements not directly tested Modify FB run Evaluate untrained results Baseline run doc[1]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[3] doc[8]/bdy[1]/article[3] Mark in top results relevant

9 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (1) 1. Standard text IR: freezing known results at the top  independent results assumption 2. New approach: remove known results+X from the collection  resColl-result : remove results only (~doc retrieval)  resColl-desc : remove results+descendants  resColl-anc : remove results+ancestors  resColl-path : remove results+desc+anc  resColl-doc : remove whole doc with known results

10 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Freezing: Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4]

11 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

12 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[3] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

13 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

14 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (2) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] block top-3 Feedback run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] Freezing:

15 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

16 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

17 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

18 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[2]/bdy[1] doc[4]/bdy[1]/ article[4] resColl-path:

19 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[4]/bdy[1]/ article[4] resColl-path:

20 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Evaluation Methodology (3) Baseline run doc[7]/bdy[1] doc[3] doc[2]/bdy[1] doc[8]/bdy[1]/article[3] doc[4]/bdy[1]/ article[1]/ sec[6] Feedback run doc[2]/bdy[1]/article[1] doc[9] doc[4]/bdy[1]/article[2] doc[4]/bdy[1]/ article[4] resColl-path:

21 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Best Evaluation Methodology? sec „ The IR process is composed …“ article body sec subsec „ For small collections …“ frontmatterbackmatter sec subsec p p P „ Figure 1 outlines …“ author „ Ian Ruthven “ citation „ D. Harman “ resColl-path

22 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Testing Evaluated Results Standard method: average – problems: Topic-id205280307325341400Avg. Baseline0.20.30.1 0.20.30.2 Modified feedback 0.2 0.10.90.2 0.3 t-test & Wilcoxon signed-rank test: gives probability p that the baseline run is better than the feedback run experiment significant if p<0.05 or p<0.01

23 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Results (1) Evaluation mode: resColl-path Feedback fileINEX metric Abs. improv. Rel. improv. T-testWSR TopX_CO_Content.xml0.01850.01121.54670.0001 xfirm_r1_cosc3s.xml0.00280.00151.09750.00030.0023 xfirm_r1_cosc5.xml0.00260.00120.92220.00280.0422 xfirm_r1_cosc3.xml0.00250.00120.88540.00320.0441 xfirm_r1_coc3s3.xml0.0031-0.0017-0.35640.93010.9995 xfirm2_r2_cop4.xml0.0032-0.0018-0.35940.85320.9732 xfirm2_r2_cot40.xml0.0025-0.0024-0.48630.92390.9987 xfirm2_r2_cot10.xml0.0023-0.0026-0.53340.94290.9999 xfirm_r1_coc3.xml0.0014-0.0034-0.71860.99930.9999 xfirm_r1_coc10.xml0.0013-0.0035-0.72810.99890.9999

24 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Results (2) Comparison of evaluation techniques based on relative improvement w.r.t. baseline run freezingresColl- anc resColl- desc resColl- doc resColl- path resColl- res c3s TopX c3s TopXc5c3s c5 TopXc5 TopX c3 TopX = TopX_CO_Content.xml c3 = xfirm_r1_cosc3.xml c3s = xfirm_r1_cosc3s.xml c5 = xfirm_r1_cosc5.xml

25 Silvana Solomon Evaluation of RF Algorithms for XML Retrieval 27 Feb 2007 Conclusions & Future Work Evaluation based on different techniques & metrics Correct improvement measurement Not solved: comparing several systems with different output Maybe a hybrid evaluation mode

Download ppt "Evaluation of Relevance Feedback Algorithms for XML Retrieval Silvana Solomon 27 February 2007 Supervisor: Dr. Ralf Schenkel."

Similar presentations

Even More TopX: Relevance Feedback Ralf Schenkel Joint work with Osama Samodi, Martin Theobald.

Even More TopX: Relevance Feedback Ralf Schenkel Joint work with Osama Samodi, Martin Theobald.
Information Extraction Lecture 4 – Named Entity Recognition II CIS, LMU München Winter Semester 2014-2015 Dr. Alexander Fraser, CIS.

Information Extraction Lecture 4 – Named Entity Recognition II CIS, LMU München Winter Semester Dr. Alexander Fraser, CIS.
Retrieval Evaluation J. H. Wang Mar. 18, 2008. Outline Chap. 3, Retrieval Evaluation –Retrieval Performance Evaluation –Reference Collections.

Retrieval Evaluation J. H. Wang Mar. 18, Outline Chap. 3, Retrieval Evaluation –Retrieval Performance Evaluation –Reference Collections.
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.

1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.

Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.
Exercising these ideas  You have a description of each item in a small collection. (30 web sites)  Assume we are looking for information about boxers,

Exercising these ideas  You have a description of each item in a small collection. (30 web sites)  Assume we are looking for information about boxers,
Section Based Relevance Feedback Student: Nat Young Supervisor: Prof. Mark Sanderson.

Section Based Relevance Feedback Student: Nat Young Supervisor: Prof. Mark Sanderson.
Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.

Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.
GENERATING AUTOMATIC SEMANTIC ANNOTATIONS FOR RESEARCH DATASETS AYUSH SINGHAL AND JAIDEEP SRIVASTAVA CS DEPT., UNIVERSITY OF MINNESOTA, MN, USA.

GENERATING AUTOMATIC SEMANTIC ANNOTATIONS FOR RESEARCH DATASETS AYUSH SINGHAL AND JAIDEEP SRIVASTAVA CS DEPT., UNIVERSITY OF MINNESOTA, MN, USA.
1 Entity Ranking Using Wikipedia as a Pivot (CIKM 10’) Rianne Kaptein, Pavel Serdyukov, Arjen de Vries, Jaap Kamps 2010/12/14 Yu-wen,Hsu.

1 Entity Ranking Using Wikipedia as a Pivot (CIKM 10’) Rianne Kaptein, Pavel Serdyukov, Arjen de Vries, Jaap Kamps 2010/12/14 Yu-wen,Hsu.
Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)

Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)
DYNAMIC ELEMENT RETRIEVAL IN A STRUCTURED ENVIRONMENT MAYURI UMRANIKAR.

DYNAMIC ELEMENT RETRIEVAL IN A STRUCTURED ENVIRONMENT MAYURI UMRANIKAR.
1 Configurable Indexing and Ranking for XML Information Retrieval Shaorong Liu, Qinghua Zou and Wesley W. Chu UCLA Computer Science Department {sliu, zou,

1 Configurable Indexing and Ranking for XML Information Retrieval Shaorong Liu, Qinghua Zou and Wesley W. Chu UCLA Computer Science Department {sliu, zou,
Re-ranking Documents Segments To Improve Access To Relevant Content in Information Retrieval Gary Madden Applied Computational Linguistics Dublin City.

Re-ranking Documents Segments To Improve Access To Relevant Content in Information Retrieval Gary Madden Applied Computational Linguistics Dublin City.
Routing of Structured Queries in Large-Scale Distributed Systems Workshop on Large-Scale Distributed Systems for Information Retrieval ACM.

Routing of Structured Queries in Large-Scale Distributed Systems Workshop on Large-Scale Distributed Systems for Information Retrieval ACM.
Retrieval Evaluation: Precision and Recall. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity.

Retrieval Evaluation: Precision and Recall. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity.
© Anselm Spoerri Lecture 13 Housekeeping –Term Projects Evaluations –Morse, E., Lewis, M., and Olsen, K. (2002) Testing Visual Information Retrieval Methodologies.

© Anselm Spoerri Lecture 13 Housekeeping –Term Projects Evaluations –Morse, E., Lewis, M., and Olsen, K. (2002) Testing Visual Information Retrieval Methodologies.
SIMS 202 Information Organization and Retrieval Prof. Marti Hearst and Prof. Ray Larson UC Berkeley SIMS Tues/Thurs 9:30-11:00am Fall 2000.

SIMS 202 Information Organization and Retrieval Prof. Marti Hearst and Prof. Ray Larson UC Berkeley SIMS Tues/Thurs 9:30-11:00am Fall 2000.
Hybrid XML Retrieval Revisited Jovan Pehcevski PhD Candidate School of CS and IT, RMIT University

Hybrid XML Retrieval Revisited Jovan Pehcevski PhD Candidate School of CS and IT, RMIT University
Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.

Similar presentations

Ads by Google