Presentation is loading. Please wait.

Presentation is loading. Please wait.

ETISEO, Nice, May 11-12 2005 PETS International Workshops on Performance Evaluation of Tracking and Surveillance James Ferryman Computational Vision Group.

Similar presentations


Presentation on theme: "ETISEO, Nice, May 11-12 2005 PETS International Workshops on Performance Evaluation of Tracking and Surveillance James Ferryman Computational Vision Group."— Presentation transcript:

1 ETISEO, Nice, May 11-12 2005 PETS International Workshops on Performance Evaluation of Tracking and Surveillance James Ferryman Computational Vision Group Department of Computer Science The University of Reading, UK

2 ETISEO, Nice, May 11-12 2005 Supported by PETS International Workshops on Performance Evaluation of Tracking and Surveillance

3 ETISEO, Nice, May 11-12 2005 Introduction Theme - Performance Evaluation of Tracking and Surveillance Successful tracking of object motions key to visual surveillance PETS started in Grenoble, France in 2000 as satellite workshop of FG2000 Not a competition http://visualsurveillance.org ftp://pets.rdg.ac.uk

4 ETISEO, Nice, May 11-12 2005 PETS - History PETS’2000 was held at FG’2000, 31 March 2000, Grenoble, France. PETS’2001 at CVPR’01. PETS’2002 at ECCV, Copenhagen, Denmark, June 1 2002. PETS2003 at ICVS, Graz; VS-PETS at ICCV2003 PETS2004 at ECCV04 WAMOP-PETS, CO, USA (Jan 05) as part of IEEE Winter Workshop Series 2005: VS-PETS at ICCV’05

5 ETISEO, Nice, May 11-12 2005 Datasets – Example – PETS2001 Five separate sets of training and test sequences. All datasets are multiview (frame sychronised). Datasets were significantly more challenging than PETS2000 (significant lighting variation, occlusion, scene activity and use of multiview data)

6 ETISEO, Nice, May 11-12 2005 Datasets Dataset 2 Dataset 1 Dataset 3

7 ETISEO, Nice, May 11-12 2005 Dataset 1

8 ETISEO, Nice, May 11-12 2005 Dataset 2

9 ETISEO, Nice, May 11-12 2005 Dataset 4

10 ETISEO, Nice, May 11-12 2005 PETS - Prerequisites Tracking results reported –should be performed using the test sequences, but the training sequences may optionally be used if the algorithms require it (for learning etc.) –may be based on a single camera view of the scene, or using multiple view data. –can be based on the entire test sequence, or a portion of it; the images may be converted to any other format and/or subsampled. –results must be submitted in XML format.

11 ETISEO, Nice, May 11-12 2005 PETS – Workshop Overview XX contributed papers ~3 sessions: e.g. appearance-based tracking, people and vehicle tracking, multiview tracking Y invited speakers Demonstration session Overall evaluation and discussion

12 ETISEO, Nice, May 11-12 2005 Quantitative PE - XML XML provides mechanism of setting up “syntax” file in form of schema Schema used to automatically validate object tracking results For PETS’2001, two schemas were used: –low-level tracking results –high-level surveillance (understanding object motions and interactions)

13 ETISEO, Nice, May 11-12 2005 Quantitative PE - XML scene understanding with multiple cameras. -->

14 ETISEO, Nice, May 11-12 2005 Quantitative PE - XML

15 Quantitative PE - XML

16 ETISEO, Nice, May 11-12 2005 D1C1: XML output

17 ETISEO, Nice, May 11-12 2005 D1 C1 - 1

18 ETISEO, Nice, May 11-12 2005 D1 C1 - 1

19 ETISEO, Nice, May 11-12 2005 D1 C1 - 1

20 ETISEO, Nice, May 11-12 2005 D1 C1 - 1

21 ETISEO, Nice, May 11-12 2005 D1 C1 - 1

22 ETISEO, Nice, May 11-12 2005 D1 C1 - 1

23 ETISEO, Nice, May 11-12 2005 D1 C1 - 2

24 ETISEO, Nice, May 11-12 2005 D1 C1 - 2

25 ETISEO, Nice, May 11-12 2005 D1 C1 - 2

26 ETISEO, Nice, May 11-12 2005 D1 C1 - 2

27 ETISEO, Nice, May 11-12 2005 D1 C1 - 2

28 ETISEO, Nice, May 11-12 2005 D1 C1- 3

29 ETISEO, Nice, May 11-12 2005 D1 C1- 3

30 ETISEO, Nice, May 11-12 2005 D1 C1- 3

31 ETISEO, Nice, May 11-12 2005 D1 C1- 3

32 ETISEO, Nice, May 11-12 2005 D1 C1- 3

33 ETISEO, Nice, May 11-12 2005 D1 C1- 3

34 ETISEO, Nice, May 11-12 2005 D1C1: XML output 1

35 ETISEO, Nice, May 11-12 2005 D1C1: XML output 2

36 ETISEO, Nice, May 11-12 2005 D1C1: XML output 3

37 ETISEO, Nice, May 11-12 2005 D1C1: XML output 4

38 ETISEO, Nice, May 11-12 2005 D1C1: XML output 5

39 ETISEO, Nice, May 11-12 2005 Performance Evaluation Evaluation of surveillance system can be judged in a number of ways: –object detection lag –object centroid position error –object area error –track incompleteness factor –accuracy of semantics of interaction –object identity error maintenance of identity through occlusion …

40 ETISEO, Nice, May 11-12 2005 384 x 288 (JPEG) 10 fps 384 x 28825-30 fps600 MHz Dual PIII 850 MHz 384 x 288 (AVI) 384 x 2885 fps 768 x 5765 fps 384 x 2886.25 fps800 MHz PIII 320 x 24029.97 fps1 GHz PIV 5fps1.7 GHz PIV 768 X 5765fps Image Format Processing Speed Processor

41 ETISEO, Nice, May 11-12 2005 Discussion Evaluation criteria are application dependent Training data – required or not? representative examples how much? Semantics of XML schema Ground truth difficult to obtain automatic evaluation - how?

42 ETISEO, Nice, May 11-12 2005 PETS Evaluation +ve: “Mindset” – engaging the community – change of culture +ve: Repository of data (PETS01 most frequently accessed) +ve: Discussion/presentation of methodologies, metrics, tools … +ve: Filters through to conferences/published literature -ve: For workshop, choice of dataset(s) + annotation -ve: More quantitative evaluation needed

43 ETISEO, Nice, May 11-12 2005 PETS, ETISEO and the future … Online web-based evaluation service (Semi-)automatic validation of XML against ground truth Repository of algorithms (incl. “strawman”), and tabulated results (rank?) Methodology for evaluation Metrics More challenging datasets (e.g. multiview) Live workshop sessions on “unseen” data Expectation that ETISEO will support PETS

44 ETISEO, Nice, May 11-12 2005 PETS’05 ICCV ’05, Beijing, China 15-16 October 2005 http://www.cbsr.ia.ac.cn/conferences/ VS-PETS-2005 http://visualsurveillance.org/PETS2005


Download ppt "ETISEO, Nice, May 11-12 2005 PETS International Workshops on Performance Evaluation of Tracking and Surveillance James Ferryman Computational Vision Group."

Similar presentations


Ads by Google