1. E VENT D ETECTION U SING AN A TTENTION -B ASED T RACKER 22 Outubro 2007. Universidade Federal do Paraná. Gerald Dalley, Xiaogang Wang, and W. Eric L. Grimson

2.  Glasgow Airport  4 cameras  9 video clips  1 for training  8 for testing Dataset Description 2

3.  Loitering  Staying in the scene for > 1 minute  Theft  Taking someone else’s baggage  Left luggage  Leaving the scene without your luggage  Evaluation  Correct event times  Correct 3D locations  Without false detections Challenge Problem 3

4.  Basic tracking first  Detect interesting tracks  Improve interesting tracks to detect events Our Approach 4

5. Processing Pipeline 5 Tracking & Event Detection Input Data Background Subtraction Background Modeling Background Modeling Illumination Normalization Illumination Normalization

6. Illumination Changes 6 BACKGROUND 126 999 2999 S08

7. Illumination Changes by Clip 7

8. Background Modeling 8 Tracking & Event Detection Input Data Background Subtraction Robust Gaussian Fit Ground-Plane Registration Background Model Search Normalized BACKGROUND Normalized S01-S08 Background Modeling Background Modeling Background Models Illumination Normalization

9.  Fundamental assumption  foreground is rare at every pixel  Reality for PETS 2007…  background is rare for the pixels we care about most  Some pixels: foreground as much as 90% of the time Breaking Adaptive Background Subtraction 9 0 0 2000 3000 4000

10. Robust Gaussian Fit (per pixel) 10  BACKGROUND clip  Foreground is rare everywhere  Fit a Gaussian  Refit to inliers

11. Need for Model Adaptation 11  Another clip (S02)  BACKGROUND’s model: Suboptimal fit BACKGROUND’s Gaussian model Background samples Foreground samples BACKGROUND’s Gaussian model Background samples Foreground samples

12. Model Adaptation 12  Until convergence  Find inliers  Shift Gaussian center

13. Improvement 13  Adaptation  Is robust  Improves FG/BG classification rates Before adaptation After adaptation Before adaptation After adaptation

14. Background Subtraction 14 Tracking & Event Detection Input Data Background Subtraction Background Modeling Illumination Normalization MRF blobs Blob Extraction Mahalanobis Distance Background Models Normalized S01-S08

15. Background Subtraction 15 Frame 2000 With Foreground Mask Foreground Mask Mahalanobis Distances Adapted Model Normalized - - = = MRF

16. Tracking & Event Detection 16 Tracking & Event Detection Input Data Background Subtraction Background Modeling Illumination Normalization alerts blobs Kalman Tracking Meanshift Tracking Event Detection

17.  Idea: Focus on tracking what we care about.  Loitering humans  Dropped luggage that becomes dissociated from its owner  Kalman tracking  Constant velocity  Low false positive rate Blob Tracking 17

18.  Loitering humans  Remain in the scene for a long time  Likely to create isolated tracks  Dispossessed luggage  Likely to create at least an isolated blob detection Detecting Humans and Luggage 18 Object Type Min. Blob Area (% of frame) Max. Blob Area (% of frame) Min. Blob Track Length humans1.5%3.0%16s luggage0.2%1.0%1 frame

19.  Blob tracking  Yields high-quality tracks (good)  Requires isolated blobs (bad)  Meanshift tracker  Learn a model (color histogram) from the good blob tracks  Tracks through occlusions  Humans  Find scene entry/exit times  Luggage  Find drop/pickup times  Associate with human owners Mean-Shift Tracking 19

20. Results 20

21.  No events occur  None detected S00 – No Defined Behavior 21

22.  Staged loitering  5.1s late S01 – General Loitering 1 (Easy)

23.  Staged loitering  1.4s late S02 – General Loitering 2 (Hard) 23

24.  Staged loitering  8.2s late  Dropped luggage  Should not trigger an alarm  No alarm triggered  Staged loitering man near purple-outlined woman  Missed  Unscripted loitering  Detected S03 – Bag Swap 1 (Easy) 24

25. S04 – Bag Swap 2 (Hard) 25 Stay close to each other the whole time

26. S05 – Theft 1 (Easy) 26  Victim enters  19.2s late  Luggage stolen  0.08s late  Thief exits  0.08s late

27. S06 – Theft 2 (Hard) 27 Thief Luggage (never isolated) Victims Assistant thief

28. S07 – Left Luggage 1 (Easy) 28  Luggage dropped  0.12s late  Owner tracked  Luggage taken  0.08s late  By owner

29. S08 – Left Luggage 2 (Hard) 29 Owner drops: 0.2s late Owner picks up: 0.12s late

30.  PETS 2007  Problem description, datasets, etc.  http://www.pets2007.net http://www.pets2007.net  My Website  The paper  http://people.csail.mit.edu/dalleyg http://people.csail.mit.edu/dalleyg Websites 30

31. Perguntas 31

32. Illumination Normalization 32 where

33. Region-of-Interest Masks 33 BG-S04 S06 S05 S07-S08

34. Background Subtraction 34 Tracking & Event Detection Input Data Background Subtraction Background Modeling Illumination Normalization BG-Biased MRF FG-Biased MRF Blob Extraction fragmented blobs merged blobs Blob Extraction Mahalanobis Distance Background Models Normalized S01-S08

35. Dual Background Subtraction 35  Low fragmentation  But blobs merged  Good for human tracking Mahalanobis Distance Map Background-Biased Blobs Foreground-Biased Blobs  Sharp boundaries  But fragmented blobs  Good for dropped luggage detection

36. Tracking & Event Detection 36 Tracking & Event Detection Input Data Background Subtraction Background Modeling Illumination Normalization alerts fragmented blobs merged blobs Kalman Tracking Meanshift Tracking Event Detection

37.  Luggage:  it often doesn’t travel far from the owner, so we need BG-biased to avoid merging the dropped luggage blob with the owner  The floor is more boring (except for specularities), so camouflaging doesn’t occur much there, relative to the vertical surfaces  Easy to tell people fragments from luggage: small people fragments move  They move when they’re isolated  They move before and after isolation  Humans  With the busyness of the scene, a BG-biased MRF produces a lot of fragments and many-to-many blob matching quickly becomes impractical  A FG-biased MRF avoids the fragmentation issue but merges lots of blobs  We only care about loiterers  Loiterers are in the scene for a long time  They’re likely to be isolated from other people at least at some point in time Why Dual Trackers/Motion Blobs 37

38. Oriented Ellipsoids 38