1. CLEF 2008 Multilingual Question Answering Track UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner

2. 2 QA 2008 Task and Exercises  QA Main task (6th edition) Pilot: QA WSD, English newswire collections with Word Sense Disambiguation  Answer Validation Exercise – AVE (3rd edition)  QA on Speech Transcripts – QAST (2nd edition)

3. 3 Main Task QA 2008 Organizing Committee  CELCT (D. Giampiccolo, P. Forner): Italian  UNED (A. Peñas): Spanish  U. Groeningen (G. Bosma): Dutch  U. Limerick (R. Sutcliff): English  DFKI (B. Sacalenau): German  ELDA/ELRA (N. Moreau): French  Linguateca (P. Rocha): Portuguese  Bulgarian Academy of Sciences (P. Osenova): Bulgarian ♦ IASI (C. Forascu): Romanian ♦ U. Basque Country (I. Alegria): Basque ♦ ILSP (P.Prokopidis): Greek

4. 4 Evolution of the Track 200320042005200620072008 Target languages 37891011 Collections News 1994+News 1995+Wikipedia Nov. 2006 Type of questions 200 Factoid + Temporal restrictions + Definitions - Type of question + Lists + Linked questions + Closed lists Supporting information Doc.Snippet Pilots and Exercises Temporal restrictions Lists AVE Real Time WiQA AVE QAST AVE QAST WSDQA

5. 5 200 questions  FACTOID  ( loc, mea, org, per, tim, cnt, obj, oth )  DEFINITION  (per, org, obj, oth)  CLOSED LIST  Who were the components of The Beatles?  Who were the last three presidents of Italy?  LINKED QUESTIONS Who was called the “Iron-Chancellor”? When was he born? Who was his first wife? ♦ Temporal restrictions by date, by period, by event ♦ NIL questions (without known answer in the collection)

6. 6 43 Activated Language Combinations (at least one registered participant) TARGET LANGUAGES (corpus and answers) BGDEELENESEUFRITNLPTRO SOURCE LANGUAGES (questions) BG DE EL EN ES EU FR IT NL PT RO

7. 7 7 Activated Tasks MONOLINGUALCROSS-LINGUALTOTAL CLEF 2003358 CLEF 200461319 CLEF 200581523 CLEF 2006 71724 CLEF 200782937 CLEF 2008103343

8. 8 8 Submitted runs MonolingualCross-lingual CLEF 200317 611 CLEF 200448 (+182%) 2028 CLEF 200567 (+40%) 4324 CLEF 200677 (+15%) 4235 CLEF 200737 (-52%) 2017 CLEF 200851 (+38%) 3120

9. 9 Participant groups NewcomersVeteransTOTALRegistered CLEF 2003--8- CLEF 2004135 18 (+125%) 22 CLEF 2005915 24 (+33%) 27 CLEF 20061020 30 (+25%) 36 CLEF 2007814 22 (-26%) 29 CLEF 20088132133

10. 10 List of Participants (random order) Bulgaria

11. 11 Groups per year and target collection Task Change Natural selection? Above 20 groups

12. 12 Groups per target collection

13. 13 2008 participation: Comparative evaluation? Lack from evaluation perspective: 4 languages without comparison between different groups Breakout session LanguageRuns Different groups Portuguese96 Spanish104 English54 German113 Romanian42 Dutch41 Basque41 French31 Bulgarian11 Italian00 Greek00

14. 14 Results: Best and Average scores

15. 15 Best scores by language

16. 16 Best scores by participant

17. 17 Results depend on type of questions  Definitions Almost solved for several systems 80%-95%  Factoids 50%-65% for several systems  Temporal restrictions Same level of difficulty as factoids for some systems  Closed lists Still very difficult  Linked questions Still very difficult  Now wikipedia provides more answers

18. 18 Conclusion  Same task as 2007 Same level of participation (slightly better)  11 target languages (9 with participation)  43 activated subtasks  21 participants  51 runs Same results (slightly better)

19. 19 Future direction  Less participants per language Poor comparison Change methodology: one task for all  Critics to QA over wikipedia Easier to find questions with IR No user model Change collection  QA proposal for 2009 SC and breakout