ETISEO, Nice, May PETS International Workshops on Performance Evaluation of Tracking and Surveillance James Ferryman Computational Vision Group Department of Computer Science The University of Reading, UK

ETISEO, Nice, May Supported by PETS International Workshops on Performance Evaluation of Tracking and Surveillance

ETISEO, Nice, May Introduction Theme - Performance Evaluation of Tracking and Surveillance Successful tracking of object motions key to visual surveillance PETS started in Grenoble, France in 2000 as satellite workshop of FG2000 Not a competition ftp://pets.rdg.ac.uk

ETISEO, Nice, May PETS - History PETS’2000 was held at FG’2000, 31 March 2000, Grenoble, France. PETS’2001 at CVPR’01. PETS’2002 at ECCV, Copenhagen, Denmark, June PETS2003 at ICVS, Graz; VS-PETS at ICCV2003 PETS2004 at ECCV04 WAMOP-PETS, CO, USA (Jan 05) as part of IEEE Winter Workshop Series 2005: VS-PETS at ICCV’05

ETISEO, Nice, May Datasets – Example – PETS2001 Five separate sets of training and test sequences. All datasets are multiview (frame sychronised). Datasets were significantly more challenging than PETS2000 (significant lighting variation, occlusion, scene activity and use of multiview data)

ETISEO, Nice, May Datasets Dataset 2 Dataset 1 Dataset 3

ETISEO, Nice, May Dataset 1

ETISEO, Nice, May Dataset 2

ETISEO, Nice, May Dataset 4

ETISEO, Nice, May PETS - Prerequisites Tracking results reported –should be performed using the test sequences, but the training sequences may optionally be used if the algorithms require it (for learning etc.) –may be based on a single camera view of the scene, or using multiple view data. –can be based on the entire test sequence, or a portion of it; the images may be converted to any other format and/or subsampled. –results must be submitted in XML format.

ETISEO, Nice, May PETS – Workshop Overview XX contributed papers ~3 sessions: e.g. appearance-based tracking, people and vehicle tracking, multiview tracking Y invited speakers Demonstration session Overall evaluation and discussion

ETISEO, Nice, May Quantitative PE - XML XML provides mechanism of setting up “syntax” file in form of schema Schema used to automatically validate object tracking results For PETS’2001, two schemas were used: –low-level tracking results –high-level surveillance (understanding object motions and interactions)

ETISEO, Nice, May Quantitative PE - XML scene understanding with multiple cameras. --> <people_tracker xmlns=" xmlns:xsi=" xsi:schemaLocation= "

ETISEO, Nice, May Quantitative PE - XML <software name="Reading People Tracker" platform="Linux" version="0.03" run_date="12/07/00"> <object_detector name="Reading People Tracker" platform="Linux" version="0.03" run_date="08/06/01"/>

Quantitative PE - XML <!-- a "target" is any object which moves or may move, usually a person, group of people, or a vehicle. The target's id is GLOBAL to all the cameras defined in "list_cameras" --> <!-- start_frame and end_frame indicate when the target has been tracked. end_frame may be unknown because it is in the future; in this case the longest known time where the object was tracked will be given --> <!-- the status of a graph node explains how the node of current target has been created or tracked. The following values may be used and added together as appropriate: 0 : default value, already tracked 1 : new track (id did not exist before) 2 : re-appearing object (id copied from last occurrence) 4 : merging (more than one parent in graph) 8 : splitting (at least one parent in graph has more than 1 child) 16 : lost (object NOT found in current image, given position etc are estimates (if available) or previous values) 32 : out of field of view (tracked object not "visible" as per definition (see elsewhere)) --> <!-- location values are defined as the sum of the following: 0 : undefined 1 : roadway 2 : in close proximity to vehicle (parking lot) 4 : on grass/verge 8 : other -->

ETISEO, Nice, May D1C1: XML output

ETISEO, Nice, May D1 C1 - 1

ETISEO, Nice, May D1 C1 - 1

ETISEO, Nice, May D1 C1 - 1

ETISEO, Nice, May D1 C1 - 1

ETISEO, Nice, May D1 C1 - 1

ETISEO, Nice, May D1 C1 - 1

ETISEO, Nice, May D1 C1 - 2

ETISEO, Nice, May D1 C1 - 2

ETISEO, Nice, May D1 C1 - 2

ETISEO, Nice, May D1 C1 - 2

ETISEO, Nice, May D1 C1 - 2

ETISEO, Nice, May D1 C1- 3

ETISEO, Nice, May D1 C1- 3

ETISEO, Nice, May D1 C1- 3

ETISEO, Nice, May D1 C1- 3

ETISEO, Nice, May D1 C1- 3

ETISEO, Nice, May D1 C1- 3

ETISEO, Nice, May D1C1: XML output 1

ETISEO, Nice, May D1C1: XML output 2

ETISEO, Nice, May D1C1: XML output 3

ETISEO, Nice, May D1C1: XML output 4

ETISEO, Nice, May D1C1: XML output 5

ETISEO, Nice, May Performance Evaluation Evaluation of surveillance system can be judged in a number of ways: –object detection lag –object centroid position error –object area error –track incompleteness factor –accuracy of semantics of interaction –object identity error maintenance of identity through occlusion …

ETISEO, Nice, May x 288 (JPEG) 10 fps 384 x fps600 MHz Dual PIII 850 MHz 384 x 288 (AVI) 384 x 2885 fps 768 x 5765 fps 384 x fps800 MHz PIII 320 x fps1 GHz PIV 5fps1.7 GHz PIV 768 X 5765fps Image Format Processing Speed Processor

ETISEO, Nice, May Discussion Evaluation criteria are application dependent Training data – required or not? representative examples how much? Semantics of XML schema Ground truth difficult to obtain automatic evaluation - how?

ETISEO, Nice, May PETS Evaluation +ve: “Mindset” – engaging the community – change of culture +ve: Repository of data (PETS01 most frequently accessed) +ve: Discussion/presentation of methodologies, metrics, tools … +ve: Filters through to conferences/published literature -ve: For workshop, choice of dataset(s) + annotation -ve: More quantitative evaluation needed

ETISEO, Nice, May PETS, ETISEO and the future … Online web-based evaluation service (Semi-)automatic validation of XML against ground truth Repository of algorithms (incl. “strawman”), and tabulated results (rank?) Methodology for evaluation Metrics More challenging datasets (e.g. multiview) Live workshop sessions on “unseen” data Expectation that ETISEO will support PETS

ETISEO, Nice, May PETS’05 ICCV ’05, Beijing, China October VS-PETS