Presentation on theme: "TrebleCLEF Evaluation package and language resources CLEF Evaluation Packages Nicolas Moreau / Khalid Choukri - ELDA."— Presentation transcript:
TrebleCLEF Evaluation package and language resources CLEF Evaluation Packages Nicolas Moreau / Khalid Choukri - ELDA
TrebleCLEF and LRs CLEF evaluation campaigns produced large volumes of valuable data A major objective of TrebleCLEF: make CLEF collections publicly available ELDA responsible for: Inventory of CLEF resources IPR issues Production of CLEF Evaluation Packages
Evaluation Packages CLEF test suites consist of: Multilingual document collections Sets of topics Sets of relevance assessments Scoring tools Participant results and related publications Step-by-step documentation on how to perform a system evaluation Reusable test suites for system benchmarking
CLEF Packages Already distributed (ELRA/ELDA catalogue): CLEF 2000-2003 (AdHoc on news + domain specific) Produced or being produced: CLEF AdHoc/News 2004-2008 => Preliminary version CLEF AdHoc/TEL 2008 CLEF QA 2003-2008 + CLEF QAST 2007-2008 CLEF Domain specific 2004-2008 GeoCLEF 2005-2008 Still in discussion: ImageCLEF, VideoCLEF WebCLEF, CLEF-IP, and others...
Distribution and Pricing Policy Distribution ELRA/ELDA catalogue: http://catalog.elra.info/ Restricted to evaluation use Pricing policy Low prices for academics Cover the management of data (catalogue management, DVD shipping, etc.)