ETISEO Benoît GEORIS, François BREMOND and Monique THONNAT ORION Team, INRIA Sophia Antipolis, France Nice, May th 2005

2 Fair Evaluation (1/4) Unbiased and transparent evaluation protocol Large participation Meaningful evaluation

3 Fair Evaluation (2/4) Unbiased and transparent evaluation protocol  Each participant should have the same effective time for testing and evaluation  Main decision should be collegiate : common agreement on videos, ground truth and metrics  Everybody keeps control on result diffusion

4 Fair Evaluation (3/4) Large participation: what people want  Meet partner expectation: large variety of videos  Need minimum efforts for tuning and testing: small video sets Large participation: what can we propose  Possibility to adapt algorithm experimentation (having a mask, choosing options such as shadows, low contrast, wind…)  Enabling a graduation of difficulties from easy to hard  Minimizing overhead for result creation (format, data exchange,…)

5 Fair Evaluation (4/4) Meaningful evaluation  Clear visualization (straightforward metrics and graphical result presentation)  For each partner, presentation of results for problems (one after one) that will be studied by ETISEO with variations from easy to hard  For a specific problem, global comparison of partner performances (e.g., sensibility of event recognition wrt image resolution)

6 Video Selection (1/4) New or already published videos ? Contextual information associated to videos Video characterization

7 Video Selection (2/4) New or already published videos ?  Advantages of new videos:  Fair, since available time is the same for everyone  Dedicated to specific problems, with graduations of difficulties  Advantages of old videos:  Easy because they are available and many people have already tested them  Enable to compare with algorithms outside ETISEO project  Mix solution  Using both old and new videos  Sharing videos with other ongoing evaluation programs

8 Video Selection (3/4) Contextual information associated to videos  3D empty scene model  Minimum information : few 3D distances drawn on the image  Maximum information: 3D scene model made available  Camera calibration  Set of 2D and 3D points  Calibration matrix taking into account distorsion or not  3D models of objects of interest

9 Video Selection (4/4) Video characterization  Partners should specify what they want and what they don’t among the 30 possibilities  From what partners want, we can select x (5?) problems (e.g., dynamic occlusion) to be studied and generate a series of y (10?) videos  From what partners do not want, we should provide tools to prevent disturbation of other simultaneous problems (e.g., wind while occlusion)

10 Data terminology (1/2) Video sequences, video clips and scenes Blobs, moving regions, physical objects of interest and contextual objects Ground truth, annotation and reference data Criteria and metrics

11 Data terminology (2/2) Definition of video analysis tasks  1) Detection of physical objects of interest  2) Classification of physical objects of interest  3) Tracking of physical objects of interest  4) Event recognition Delimitation of tasks to be evaluated  Different types of combination (only task 1 vs combined task 1&2)  Evaluation of each task whatever the combination

12 Ground truth and Metrics For each task, definition of ground truth and metrics Annotation tool (Viper, Reading tool,…) Format for ground truth (XML, MPEG7,…)

13 MPEG7 Example T12:55:00:25F1000 PT0H1M5S725N1000F T12:55:00:25F1000 PT0H1M5S725N1000F T12:55:00:25F T12:55:00:75F T12:55:00:125F

14 XML Example

15 Ground Truth Definition : example