Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cross-Language Evaluation Forum (CLEF) IST-2000-31002 Expected Kick-off Date: August 2001 Carol Peters IEI-CNR, Pisa, Italy Carol Peters: blabla Carol.

Similar presentations


Presentation on theme: "Cross-Language Evaluation Forum (CLEF) IST-2000-31002 Expected Kick-off Date: August 2001 Carol Peters IEI-CNR, Pisa, Italy Carol Peters: blabla Carol."— Presentation transcript:

1 Cross-Language Evaluation Forum (CLEF) IST-2000-31002 Expected Kick-off Date: August 2001 Carol Peters IEI-CNR, Pisa, Italy Carol Peters: blabla Carol Peters: blabla

2 Concertation Event, Vienna, 21 June 2001 2 Cross-Language Evaluation Forum Objectives Promote research in cross-language system development for European languages by providing an appropriate infrastructure for: zsystem evaluation, testing and tuning zcomparison and discussion of results between R&D groups working on common problems zbuilding test-suites for cross-language system developers

3 Concertation Event, Vienna, 21 June 2001 3 Evaluation for Cross- Language Systems Why Evaluation is Important for CLIR zCLIR systems are still in experimental stage of development  Evaluation activities stimulate progress through objective assessment and also by comparison of systems and approaches

4 Concertation Event, Vienna, 21 June 2001 4 Evaluation for Cross- Language Systems yevaluation methodolgy yreference multilingual document collection ystatements of information needs (> queries) in multiple languages yobjective assessment of results ycomparative analysis of results Creating the infrastructure for an evaluation campaign

5 Concertation Event, Vienna, 21 June 2001 5 Cross-Language Evaluation Forum Background zJan. 2000 - CLEF launched as collaboration between DELOS NoE and US National Institute for Standards and Technology (NIST) and the TREC Conferences yMethodology for CLEF is an adaptation of TREC evaluation methodology for multilingual context zCLEF 2000 and 2001 organised within DELOS zFrom August 2001, CLEF becomes independent

6 Concertation Event, Vienna, 21 June 2001 6 CLEF 2001 Task Description Four main evaluation tracks in CLEF 2001: zmultilingual information retrieval zbilingual information retrieval zmonolingual (non-English) information retrieval zdomain-specific IR plus zexperimental track for interactive C-L systems

7 Concertation Event, Vienna, 21 June 2001 7 CLEF 2001 Multilingual Data Collection zMultilingual comparable corpus of news agencies and newspaper documents for six languages (DE,EN,FR,IT,NL,SP). Over 1 million documents zCommon set of 50 topics (from which queries are extracted) created in 9 European languages (DE,EN,FR,IT,NL,SP+FI,RU,SV) and 3 Asian languages (JP,TH,ZH)

8 Concertation Event, Vienna, 21 June 2001 8 Topics either DE,E,F,I or FI,NL,SP,SV EnglishGermanFrenchItalian Participant’s MLIR/CLIR Information Retrieval System documents CLEF 2001 Multilingual IR One result list of DE, FE, F and I documents ranked in decreasing order of estimated relevance

9 Concertation Event, Vienna, 21 June 2001 9 CLEF 2001 Bilingual IR Task : query language DE,FR,IT,FI,NL,SP,SV, RU,ZH,JP,TH - target document collection is English Goal: retrieve documents for target language, listing results in ranked list zEasier task for beginners !

10 Concertation Event, Vienna, 21 June 2001 10 CLEF 2001 Monolingual IR Task: querying document collections in FR|DE|IT|NL|SP Goal: acquire better understanding of language dependent retrieval problems zdifferent languages present different retrieval problems zissues include word order, morphology, diacritic characters, language variants

11 Concertation Event, Vienna, 21 June 2001 11 CLEF 2001 Domain-Specific IR Task: querying a structured database from a vertical domain (social sciences) in German zGerman/English/Russian thesaurus and English translations of document titles  Monolingual (DE) or cross-language (DE, EN, RU) task

12 Concertation Event, Vienna, 21 June 2001 12 CLEF 2001 Participation z30 groups: 8 N.American; 18 European; 4 Rest of the World zRuns submitted for all tasks: yCross-Language = 20 groups xMultilingual = 8 groups xBilingual -> EN= 18 groups xBilingual -> NL= 3 groups yMonolingual = 20 groups yDomain-specific = 1 group zA total of approx 200 runs were submitted

13 Concertation Event, Vienna, 21 June 2001 13 Approaches to CLIR CLEF 2000 zcommercial MT systems (Systran, Lernout and Hauspie Power Translator) zbilingual dictionary look-up zaligned parallel corpora (web-derived) zsimilarity thesaurus (using comparable corpora) Different strategies experimented for query expansion and results merging

14 Concertation Event, Vienna, 21 June 2001 14 Evaluation - Summing up zsystem evaluation is not a competition to find the best zevaluation provides opportunity to test, tune, and compare approaches in order to improve system performance zan evaluation campaign creates a community interested in examining the same issues and comparing ideas and experiences

15 Concertation Event, Vienna, 21 June 2001 15 Cross-Language Evaluation Forum zIntentions for CLEF 2002/2003 zstudy evaluation methodologies wrt user needs zaddition of more languages zaddition of new tasks (eg interactive CLEF) zC-L evaluation for other document types (eg speech) zproduce CLIR system test-suites for the R&D community

16 Concertation Event, Vienna, 21 June 2001 16 Cross-Language Evaluation Forum z For more information: z http://www.clef-campaign.org or carol@iei.pi.cnr.it


Download ppt "Cross-Language Evaluation Forum (CLEF) IST-2000-31002 Expected Kick-off Date: August 2001 Carol Peters IEI-CNR, Pisa, Italy Carol Peters: blabla Carol."

Similar presentations


Ads by Google