1. CLEF 2009 - Kerkyra Robust – Word Sense Disambiguation exercise UBC: Eneko Agirre, Arantxa Otegi UNIPD: Giorgio Di Nunzio UH: Thomas Mandl

2. CLEF 2009 - Kerkyra2 Introduction Robust: emphasize difficult topics using non-linear combination of topic results (GMAP) WSD: also automatic word sense annotation: English documents and topics (English WordNet) Spanish topics (Spanish WordNet - closely linked to the English WordNet) Participants explore how the word senses (plus the semantic information in wordnets) can be used in IR and CLIR This is the second edition of Robust-WSD

3. CLEF 2009 - Kerkyra3 Documents News collection: LA Times 94, Glasgow Herald 95 Sense information added to all content words Lemma Part of speech Weight of each sense in WordNet 1.6 XML with DTD provided Two leading WSD systems: National University of Singapore University of the Basque Country Significant effort (100Mword corpus) Special thanks to Hwee Tou Ng and colleagues from NUS and Oier Lopez de Lacalle from UBC

4. CLEF 2009 - Kerkyra4 Documents: example XML

5. CLEF 2009 - Kerkyra5 Topics We used existing CLEF topics in English and Spanish: 2001; 41-90; LA 94 2002; 91-140; LA 94 2004; 201-250; GH 95 2003; 141-200; LA 94, GH 95 2005; 251-300; LA 94, GH 95 2006; 301-350; LA 94, GH 95 First three as training (plus relevance judg.) Last three for testing

6. CLEF 2009 - Kerkyra6 Topics: WSD English topics were disambiguated by both NUS and UBC systems Spanish topics: no large-scale WSD system available, so we used the first-sense heuristic Word sense codes are shared between Spanish and English wordnets Sense information added to all content words Lemma Part of speech Weight of each sense in WordNet 1.6 XML with DTD provided

7. CLEF 2009 - Kerkyra7 Topics: WSD example

8. CLEF 2009 - Kerkyra8 Evaluation Reused relevance assessments from previous years Relevance assessment for training topics were provided alongside the training topics MAP and GMAP Participants had to send at least one run which did not use WSD and one run which used WSD

9. CLEF 2009 - Kerkyra9 Participation 10 official participants 58 monolingual runs 31 bilingual runs MonolingualBilingual AlicanteX DarmstadtX GenevaXX IxaXX JaenX Know-centerXX ReinaXX UfrgsXX UnibaXX ValenciaX

10. CLEF 2009 - Kerkyra10 Monolingual results MAP: non-WSD best, 2 participants improve using WSD GMAP: non-WSD best, 3 participants improve using WSD TrackParticipantMAPGMAPΔMAPΔGMAP English 1darmstadt 45.09 20.42-- 2reina44.52 21.18 -- 3uniba42.5017.93-- 4geneva41.7117.88-- 5know-center41.7018.64-- English WSD 1darmstadt45.0020.49-0.09+0.07 2uniba43.4619.60+0.96+1.67 3know-center42.2219.47+0.52+0.87 4reina41.2318.38-1.27-2.80 5geneva38.1116.26-3.59-2.38

11. CLEF 2009 - Kerkyra11 Monolingual: using WSD Darmstadt: combination of several indexes, including monolingual translation model No improvement using WSD Reina: UNINE: synset indexes, combine with results from other indexes Improvement in GMAP UCM: query expansion using structured queries Improvement in MAP and GMAP IXA: use semantic relatedness to expand documents No improvement using WSD GENEVA: synset indexes, expanding to synonyms and hypernyms No improvement, except for some topics UFRGS: only use lemmas (plus multiwords) Improvement in MAP and GMAP

12. CLEF 2009 - Kerkyra12 Monolingual: using WSD UNIBA: combine synset indexes (best sense) Improvements in MAP Univ. of Alicante: expand to all synonyms of best sense Improvement on train / decrease on test Univ. of Jaen: combine synset indexes (best sense) No improvement, except for some topics

13. CLEF 2009 - Kerkyra13 Bilingual results MAP and GMAP: best results for non-WSD 2 participants increase GMAP using WSD, 2 increase MAP. Improvements are rather small. TrackParticipantMAPGMAPΔMAPΔGMAP Es-En 1reina 38.42 15.11-- 2uniba38.0913.11-- 3know-center28.9806.79-- 4ufrgs27.6507.37-- 5Ixa18.0501.90-- Es-En WSD 1uniba37.5313.82-0.56+0.71 2geneva36.6316.02-- 3reina30.3209.38-8.10-5.73 4know-center29.6407.05+0.66+0.26 5ixa18.3801.98+0.33+0.08

14. CLEF 2009 - Kerkyra14 Bilingual: using WSD IXA: wordnets as the sole sources for translation Improvement in MAP UNIGE: translation of topic for baseline No improvement UFRGS: association rules from parallel corpora, plus use of lemmas (no WSD) No improvement UNIBA: wordnets as the sole sources for translation Improvement in both MAP and GMAP

15. CLEF 2009 - Kerkyra15 Conclusions and future Successful participation 10 participants Use of word senses allows small improvements on some stop scoring systems Further analysis ongoing: Manual analysis of topics which get significant improvement with WSD Significance tests (WSD non-WSD) No need of another round: All necessary material freely available http://ixa2.si.ehu.es/clirwsd Topics, documents (no word order, Lucene indexes), relevance assesments, WSD tags

16. CLEF 2009 - Kerkyra Robust – Word Sense Disambiguation exercise Thank you!

17. CLEF 2009 - Kerkyra17

18. CLEF 2009 - Kerkyra18 Word senses can help CLIR We will provide state-of-the-art WSD tags For the first time we offer sense-disambiguated collection All senses with confidence scores (error propag.) The participant can choose how to use it (e.g. nouns only) Also provide synonyms/translations for senses The disambiguated collection allows for: Expanding the collection to synonyms and broader terms Translation to all languages that have a wordnet Focused expansion/translation of collection Higher recall Sense-based blind relevance feedback There is more information in the documents

19. CLEF 2009 - Kerkyra19 CLIR WSD exercise Add the WSD tagged collection/topics as an additional “language” in the ad-hoc task Same topics Same document collection Just offer an additional resource An additional run: With and without WSD Tasks: X2ENG and ENG2ENG (control) Extra resources needed: Relevance assessment of the additional runs

20. CLEF 2009 - Kerkyra20 Usefulness of WSD on IR/CLIR disputed, but … Real compared to artificial experiments Expansion compared to just WSD Weighted list of senses compared to best sense Controlling which word to disambiguate WSD technology has improved Coarser-grained senses (90% acc. on Semeval 2007)

21. CLEF 2009 - Kerkyra21 QA WSD pilot exercise Add the WSD tagged collection/queries to the multilingual Q/A task Same topics LA94 GH95 (Not wikipedia) In addition to the word senses we provide: Synonyms / translations for those senses Need to send one run to the multilingual Q/A task 2 runs, with and without WSD Tasks: X2ENG and ENG2ENG (for QA WSD participants only) Extra resources needed: Relevance assessment of the additional runs

22. CLEF 2009 - Kerkyra22 QA WSD pilot exercise Details: Wikipedia won’t be disambiguated Only a subset of the main QA will be comparable In main QA, multiple answers are required In addition, to normal evaluation, evaluate first reply not coming from wikipedia

23. CLEF 2009 - Kerkyra23 WSD 4 AVE In addition to the word senses provide: Synonyms / translations for those senses Need to send two runs (one more than other part.): With and without WSD Tasks: X2ENG and ENG2ENG (control) Additional resources: Provide word sense tags to the snippets returned by QA results (automatic mapping to original doc. Collection)