1. Scientific Development Branch Dataset Production and Performance Evaluation for Event Detection and Tracking Paul Hosmer Detection and Vision Systems Group

2. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Outline Defining a requirement What to include in datasets Constraints Evaluation and Metrics Case Study

3. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Intelligent Video –Started in early 1990’s –FABIUS –Amethyst –Through to 2000’s –VMD capability study –Standards-based evaluations Background

4. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 What did we want to achieve? Test systems in short period of time Provide data and requirements to research community Dataset production Problem: what to include?

5. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Scenario definition What is an event? Where does the scenario take place? What challenges are posed by the environment? Ask end users / gauge demand Conduct capability study Monitor environment, apply a priori knowledge

6. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Scenario definition Abandoned Baggage When is an object abandoned? What types of object? Attributes of person?

7. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Scenario definition “Abandoned object” During the current clip, a person has placed an object which was in their possession when they entered the clip onto the floor or a seat in the detection area & That person has left the detection area without the object & Over sixty seconds after they left the detection area, that person has still not returned to the object & The object remains in the detection area.

8. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Scenario definition Key environmental factors: Lighting changes – film dawn and dusk Rain and snow Night – head lights and low SNR

9. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 How much data? Need to demonstrate performance on wide range of imagery Statistical significance Need large training and test corpus – 100’s of events Unseen data for verification

10. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Constraints You can’t always capture the event you want – simulation Make simulations as close to the requirement as possible Storage vs image quality – what will you want to do with the data at a later time? Cost – try to film as much variation/events as you can

11. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Performance Evaluation Importance of metrics – consistency across different evaluations When is an event detected? Real time evaluation, 10x real time, offline… which is most useful? Statistically significant unseen dataset: Performance on training data does not tell me anything useful about robustness

12. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 How HOSDB does it Simulate real analogue CCTV system ~ 215,000 frames per scenario evaluation ~ Evaluation 300 events 60s to alarm after GT alarm condition is satisfied One figure of merit for ranking

13. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 F1 score for event detection F1 = (α + 1)RP R + αP α ranges from 0.35 to 75 depending on scenario and application R = TP TP+FN P = TP TP+FP where

14. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 What about Tracking? 5 th i-LIDS scenario –Multiple Camera Tracking –Increasing interest from end users –Significant potential to enhance operator effectiveness and aid post event investigation The Problem –Unifying tracking labelling across multiple camera views Dataset and Evaluation Problem –Synchronisation

15. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Operational Requirement Camera Requirements: –Existing CCTV systems –Cameras are a mixture of overlapping and non-overlapping –Internal cameras are generally fixed and colour Scene Contents: –Scenes are likely to contain rest points –Varying traffic densities Target Description: –There may be multiple targets –Targets from wide demographic

16. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Imagery Collection

17. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Imagery Collection Location –Large Transport Hub (airport)Targets –Varied Targets –Differing target behaviour –Varying crowd densitiesEnvironment –Lighting changes –Filmed at Dawn, Day, Dusk and NightVolume –5 cameras –1.35 Million frames –Single and multiple target –1000+ target events –1TB external HDD

18. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007

19. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Dataset structure Target Event Set MCT01 MCT02 MCT03 TES Properties Daytime High density Night time Low density Etc Overlapping StageNon-Overlapping Stage Mixed Stage

20. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 Performance Metric F1 = 2RP R+P –F1 must be greater than or equal to 0.25 for the track to be a True Positive P = Overlapping Pixels Total Track Pixels R = Overlapping Pixels Total Ground Truth Pixels

21. SCIENTIFIC DEVELOPMENT BRANCH PAUL HOSMER BMVA Performance Evaluation Symposium 2007 –Performance evaluation is important –Evaluations need to use more data –With richer content –With widely accepted definitions and metrics –Demonstrate improved performance