1. ETISEO, Nice, May 11-12 2005 PETS International Workshops on Performance Evaluation of Tracking and Surveillance James Ferryman Computational Vision Group Department of Computer Science The University of Reading, UK

2. ETISEO, Nice, May 11-12 2005 Supported by PETS International Workshops on Performance Evaluation of Tracking and Surveillance

3. ETISEO, Nice, May 11-12 2005 Introduction Theme - Performance Evaluation of Tracking and Surveillance Successful tracking of object motions key to visual surveillance PETS started in Grenoble, France in 2000 as satellite workshop of FG2000 Not a competition http://visualsurveillance.org ftp://pets.rdg.ac.uk

4. ETISEO, Nice, May 11-12 2005 PETS - History PETS’2000 was held at FG’2000, 31 March 2000, Grenoble, France. PETS’2001 at CVPR’01. PETS’2002 at ECCV, Copenhagen, Denmark, June 1 2002. PETS2003 at ICVS, Graz; VS-PETS at ICCV2003 PETS2004 at ECCV04 WAMOP-PETS, CO, USA (Jan 05) as part of IEEE Winter Workshop Series 2005: VS-PETS at ICCV’05

5. ETISEO, Nice, May 11-12 2005 Datasets – Example – PETS2001 Five separate sets of training and test sequences. All datasets are multiview (frame sychronised). Datasets were significantly more challenging than PETS2000 (significant lighting variation, occlusion, scene activity and use of multiview data)

6. ETISEO, Nice, May 11-12 2005 Datasets Dataset 2 Dataset 1 Dataset 3

7. ETISEO, Nice, May 11-12 2005 Dataset 1

8. ETISEO, Nice, May 11-12 2005 Dataset 2

9. ETISEO, Nice, May 11-12 2005 Dataset 4

10. ETISEO, Nice, May 11-12 2005 PETS - Prerequisites Tracking results reported –should be performed using the test sequences, but the training sequences may optionally be used if the algorithms require it (for learning etc.) –may be based on a single camera view of the scene, or using multiple view data. –can be based on the entire test sequence, or a portion of it; the images may be converted to any other format and/or subsampled. –results must be submitted in XML format.

11. ETISEO, Nice, May 11-12 2005 PETS – Workshop Overview XX contributed papers ~3 sessions: e.g. appearance-based tracking, people and vehicle tracking, multiview tracking Y invited speakers Demonstration session Overall evaluation and discussion

12. ETISEO, Nice, May 11-12 2005 Quantitative PE - XML XML provides mechanism of setting up “syntax” file in form of schema Schema used to automatically validate object tracking results For PETS’2001, two schemas were used: –low-level tracking results –high-level surveillance (understanding object motions and interactions)

13. ETISEO, Nice, May 11-12 2005 Quantitative PE - XML scene understanding with multiple cameras. --> <people_tracker xmlns="http://www.cvg.cs.reading.ac.uk/PETS2001" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation= "http://www.cvg.cs.reading.ac.uk/PETS2001 http://www.cvg.cs.reading.ac.uk/PETS2001/XML/surveillance.xsd">

14. ETISEO, Nice, May 11-12 2005 Quantitative PE - XML <software name="Reading People Tracker" platform="Linux" version="0.03" run_date="12/07/00"> <object_detector name="Reading People Tracker" platform="Linux" version="0.03" run_date="08/06/01"/>

15. Quantitative PE - XML <!-- a "target" is any object which moves or may move, usually a person, group of people, or a vehicle. The target's id is GLOBAL to all the cameras defined in "list_cameras" --> <!-- start_frame and end_frame indicate when the target has been tracked. end_frame may be unknown because it is in the future; in this case the longest known time where the object was tracked will be given --> <!-- the status of a graph node explains how the node of current target has been created or tracked. The following values may be used and added together as appropriate: 0 : default value, already tracked 1 : new track (id did not exist before) 2 : re-appearing object (id copied from last occurrence) 4 : merging (more than one parent in graph) 8 : splitting (at least one parent in graph has more than 1 child) 16 : lost (object NOT found in current image, given position etc are estimates (if available) or previous values) 32 : out of field of view (tracked object not "visible" as per definition (see elsewhere)) --> <!-- location values are defined as the sum of the following: 0 : undefined 1 : roadway 2 : in close proximity to vehicle (parking lot) 4 : on grass/verge 8 : other -->

16. ETISEO, Nice, May 11-12 2005 D1C1: XML output

17. ETISEO, Nice, May 11-12 2005 D1 C1 - 1

18. ETISEO, Nice, May 11-12 2005 D1 C1 - 1

19. ETISEO, Nice, May 11-12 2005 D1 C1 - 1

20. ETISEO, Nice, May 11-12 2005 D1 C1 - 1

21. ETISEO, Nice, May 11-12 2005 D1 C1 - 1

22. ETISEO, Nice, May 11-12 2005 D1 C1 - 1

23. ETISEO, Nice, May 11-12 2005 D1 C1 - 2

24. ETISEO, Nice, May 11-12 2005 D1 C1 - 2

25. ETISEO, Nice, May 11-12 2005 D1 C1 - 2

26. ETISEO, Nice, May 11-12 2005 D1 C1 - 2

27. ETISEO, Nice, May 11-12 2005 D1 C1 - 2

28. ETISEO, Nice, May 11-12 2005 D1 C1- 3

29. ETISEO, Nice, May 11-12 2005 D1 C1- 3

30. ETISEO, Nice, May 11-12 2005 D1 C1- 3

31. ETISEO, Nice, May 11-12 2005 D1 C1- 3

32. ETISEO, Nice, May 11-12 2005 D1 C1- 3

33. ETISEO, Nice, May 11-12 2005 D1 C1- 3

34. ETISEO, Nice, May 11-12 2005 D1C1: XML output 1

35. ETISEO, Nice, May 11-12 2005 D1C1: XML output 2

36. ETISEO, Nice, May 11-12 2005 D1C1: XML output 3

37. ETISEO, Nice, May 11-12 2005 D1C1: XML output 4

38. ETISEO, Nice, May 11-12 2005 D1C1: XML output 5

39. ETISEO, Nice, May 11-12 2005 Performance Evaluation Evaluation of surveillance system can be judged in a number of ways: –object detection lag –object centroid position error –object area error –track incompleteness factor –accuracy of semantics of interaction –object identity error maintenance of identity through occlusion …

40. ETISEO, Nice, May 11-12 2005 384 x 288 (JPEG) 10 fps 384 x 28825-30 fps600 MHz Dual PIII 850 MHz 384 x 288 (AVI) 384 x 2885 fps 768 x 5765 fps 384 x 2886.25 fps800 MHz PIII 320 x 24029.97 fps1 GHz PIV 5fps1.7 GHz PIV 768 X 5765fps Image Format Processing Speed Processor

41. ETISEO, Nice, May 11-12 2005 Discussion Evaluation criteria are application dependent Training data – required or not? representative examples how much? Semantics of XML schema Ground truth difficult to obtain automatic evaluation - how?

42. ETISEO, Nice, May 11-12 2005 PETS Evaluation +ve: “Mindset” – engaging the community – change of culture +ve: Repository of data (PETS01 most frequently accessed) +ve: Discussion/presentation of methodologies, metrics, tools … +ve: Filters through to conferences/published literature -ve: For workshop, choice of dataset(s) + annotation -ve: More quantitative evaluation needed

43. ETISEO, Nice, May 11-12 2005 PETS, ETISEO and the future … Online web-based evaluation service (Semi-)automatic validation of XML against ground truth Repository of algorithms (incl. “strawman”), and tabulated results (rank?) Methodology for evaluation Metrics More challenging datasets (e.g. multiview) Live workshop sessions on “unseen” data Expectation that ETISEO will support PETS

44. ETISEO, Nice, May 11-12 2005 PETS’05 ICCV ’05, Beijing, China 15-16 October 2005 http://www.cbsr.ia.ac.cn/conferences/ VS-PETS-2005 http://visualsurveillance.org/PETS2005