Cross-Language Evaluation Forum (CLEF) IST Expected Kick-off Date: August 2001 Carol Peters IEI-CNR, Pisa, Italy Carol Peters: blabla Carol Peters: blabla

Concertation Event, Vienna, 21 June Cross-Language Evaluation Forum Objectives Promote research in cross-language system development for European languages by providing an appropriate infrastructure for: zsystem evaluation, testing and tuning zcomparison and discussion of results between R&D groups working on common problems zbuilding test-suites for cross-language system developers

Concertation Event, Vienna, 21 June Evaluation for Cross- Language Systems Why Evaluation is Important for CLIR zCLIR systems are still in experimental stage of development  Evaluation activities stimulate progress through objective assessment and also by comparison of systems and approaches

Concertation Event, Vienna, 21 June Evaluation for Cross- Language Systems yevaluation methodolgy yreference multilingual document collection ystatements of information needs (> queries) in multiple languages yobjective assessment of results ycomparative analysis of results Creating the infrastructure for an evaluation campaign

Concertation Event, Vienna, 21 June Cross-Language Evaluation Forum Background zJan CLEF launched as collaboration between DELOS NoE and US National Institute for Standards and Technology (NIST) and the TREC Conferences yMethodology for CLEF is an adaptation of TREC evaluation methodology for multilingual context zCLEF 2000 and 2001 organised within DELOS zFrom August 2001, CLEF becomes independent

Concertation Event, Vienna, 21 June CLEF 2001 Task Description Four main evaluation tracks in CLEF 2001: zmultilingual information retrieval zbilingual information retrieval zmonolingual (non-English) information retrieval zdomain-specific IR plus zexperimental track for interactive C-L systems

Concertation Event, Vienna, 21 June CLEF 2001 Multilingual Data Collection zMultilingual comparable corpus of news agencies and newspaper documents for six languages (DE,EN,FR,IT,NL,SP). Over 1 million documents zCommon set of 50 topics (from which queries are extracted) created in 9 European languages (DE,EN,FR,IT,NL,SP+FI,RU,SV) and 3 Asian languages (JP,TH,ZH)

Concertation Event, Vienna, 21 June Topics either DE,E,F,I or FI,NL,SP,SV EnglishGermanFrenchItalian Participant’s MLIR/CLIR Information Retrieval System documents CLEF 2001 Multilingual IR One result list of DE, FE, F and I documents ranked in decreasing order of estimated relevance

Concertation Event, Vienna, 21 June CLEF 2001 Bilingual IR Task : query language DE,FR,IT,FI,NL,SP,SV, RU,ZH,JP,TH - target document collection is English Goal: retrieve documents for target language, listing results in ranked list zEasier task for beginners !

Concertation Event, Vienna, 21 June CLEF 2001 Monolingual IR Task: querying document collections in FR|DE|IT|NL|SP Goal: acquire better understanding of language dependent retrieval problems zdifferent languages present different retrieval problems zissues include word order, morphology, diacritic characters, language variants

Concertation Event, Vienna, 21 June CLEF 2001 Domain-Specific IR Task: querying a structured database from a vertical domain (social sciences) in German zGerman/English/Russian thesaurus and English translations of document titles  Monolingual (DE) or cross-language (DE, EN, RU) task

Concertation Event, Vienna, 21 June CLEF 2001 Participation z30 groups: 8 N.American; 18 European; 4 Rest of the World zRuns submitted for all tasks: yCross-Language = 20 groups xMultilingual = 8 groups xBilingual -> EN= 18 groups xBilingual -> NL= 3 groups yMonolingual = 20 groups yDomain-specific = 1 group zA total of approx 200 runs were submitted

Concertation Event, Vienna, 21 June Approaches to CLIR CLEF 2000 zcommercial MT systems (Systran, Lernout and Hauspie Power Translator) zbilingual dictionary look-up zaligned parallel corpora (web-derived) zsimilarity thesaurus (using comparable corpora) Different strategies experimented for query expansion and results merging

Concertation Event, Vienna, 21 June Evaluation - Summing up zsystem evaluation is not a competition to find the best zevaluation provides opportunity to test, tune, and compare approaches in order to improve system performance zan evaluation campaign creates a community interested in examining the same issues and comparing ideas and experiences

Concertation Event, Vienna, 21 June Cross-Language Evaluation Forum zIntentions for CLEF 2002/2003 zstudy evaluation methodologies wrt user needs zaddition of more languages zaddition of new tasks (eg interactive CLEF) zC-L evaluation for other document types (eg speech) zproduce CLIR system test-suites for the R&D community

Concertation Event, Vienna, 21 June Cross-Language Evaluation Forum z For more information: z or