ETISEO Evaluation Nice, May th 2005 Evaluation Cycles

Basic evaluation process - Participant registration, - Ask for corpus data trough ETISEO web-site formulary, - Acceptance of limited usage of the videos. - Receive video corpus, - Process tests on its own, - Send back results to the evaluator. -The evaluator process the comparison with ground truth, - Send metric values to the participant.

3 Phases - Definition & Preparation ( 2005 ) - Metrics definition, tools creation - video sequences recording - Test cycle for validation ( 1st semester 2006 ) - realisation of an entire evaluation cycle with participants on the test data set. - ETISEO evaluation cycle ( 2rd semester 2006 ) - final ETISEO evaluation cycle. ETISEO project contains 3 main phases :

3 Seminars - 1st seminar : Nice, may community participations, - Definition process, - starting corpus creation. - 2rd seminar : INRETS, end of resources distribution, - starting test cycle with participants. - 3rd seminar : INRIA, end of evaluation results communication, - new knowlegde analysis, - closing ETISEO project.… Launch evaluation process …

Data set Three data sets are planed 1- Work data set: - Representative of the various sequences ( but non- exhaustive), - Distributed at the beginning of the collecting phase, - In order to give participants the maximum amount of time to run their algorithms on ETISEO corpus. - non evaluation process

Data set Test data set: - Representative of next evaluation data set. - Sequences illustrating several cases predefined in the evaluation process. - Distributed to the participant for test cycle : validation of the evaluation cycle. - Associated ground truth, - Metrics and automatic comparison tool validation, - Complete « test evaluation cycle », - Evaluation results comparison.

Data set Evaluation data set: - Exhaustive representation of elected topics, - Signifiant for statistical evaluation, - Contents not used in previous test cycle, - Created to assess performances of participant algorithms : ETISEO evaluation.

Evaluation Cycles 3 Seminaries 3 data sets 2 Evaluation Cycles

Video processing tasks The following tasks are considered all along the video processing chain: Task 1: Detection of physical objects of interest, Task 2: Classification of physical objects of interest, Task 3: Tracking of physical objects of interest, Task 4: Event recognition. Video data set, ground truth and metrics should enable to highlight different tasks processing to extract detailed analysis of the algorithms. Participants will evaluate tasks corresponding to their usual research topics.

To Be Define - video data set contents & format, - suitable and realistic ground truth, - metrics offering optimal added value, - participants algorithms classification, - results, ground truth format, - questionnaire associated to algorithms results, - results communication process. Several elements have to be define, during the 1 st Seminar and following :

Web Site