1. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 1 A Distributed Outdoor Video Surveillance System for Detection of Abnormal People Trajectories Simone Calderara, Rita Cucchiara, Andrea Prati Imagelab laboratory University of Modena and Reggio Emilia, Italy

2. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 2 Agenda Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling Probabilistic people trajectory classification Conclusions Motivations

3. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 3 Motivations Video surveillance: why? –Increasing security level of public places –Crime Prevention … Why automated video surveillance? –Manual monitoring of large areas is difficult –Humans typically focus their attention on particular spots (some others may not be observed) –Interesting information may be extracted and stored for subsequent analysis (posterity logging for forensic off-line analysis)

4. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 4 Motivations (2) Distributed multiple cameras mean: –Wider coverage of the scene –Redundant data (improved accuracy) –Different viewpoints disambiguate groups, help with occlusions Distributed multiple cameras - disadvantages: –System complexity can seriously rise with many cameras –Using multiple cameras is almost useless if data from different cameras are not correlated –Need for camera communication/coordination

5. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 5 Motivations (3) Previous generations of automatic surveillance systems are focused on robustly performing low-level tasks (motion detection, segmentation, perimeter control,…) Next generations will need to operate at a higher level of abstraction (infer or learn behavioral patterns, detect specific behaviors, understand what is happening in the scene,…)

6. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 6 Agenda Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling Probabilistic people trajectory classification Conclusions Motivations The IMAGELAB distributed video surveillance system

7. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 IMAGELAB VS system Sakbot Ad hoc Hecol PPC Projects and prototypes since 1998

8. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 8 Park experimentation (LAICA Project Reggio Emilia)

9. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 9 Agenda Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling Probabilistic people trajectory classification Experiments Conclusions Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking

10. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 10 Motion detection SAKBOT based on temporal median + selective knowledge-based updating Suitable modifications of SAKBOT [1] system to deal with outdoor requirements: –Bootstrapping: initial bkg model created using single difference at block-level (16x16) –Adaptive bkg differencing: hysteresis (local and pixel- varying) thresholding of the bkg difference [1] R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, “Detecting moving objects, ghosts and shadows in video streams,” IEEE Trans. on PAMI, vol. 25, no. 10, pp. 1337–1342, Oct. 2003.

11. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 11 Motion detection (2) SAKBOT’s modifications (cont’d): –Fast “ghost” suppression: similar to bootstrapping (based on single difference), but at region-level: valid object only if a sufficient number of pixels are moving –Others: object validation, shadow suppression, …

12. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 12 Object tracking Apperance-based tracking approaches keep track not only of the state but also of the shape at pixel- level, necessary for gait or posture analysis AD-HOC (Appearance Driven Human tracking with Occlusion Classification) [2] based not only on object’s position and speed, but also on its appearance map and probability mask [2] R. Vezzani, C. Grana, R. Cucchiara, “AD-HOC: Appearance Driven Human tracking with Occlusion Classification”, under review in Pattern Recognition, 2007

13. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 13 Agenda Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling Probabilistic people trajectory classification Conclusions Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling

14. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 [3] S. CALDERARA, R. CUCCHIARA, PRATI A. Bayesian-competitive Consistent Labeling for People Surveillance. In press on IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE Multi-camera consistent labeling Consistent labeling allows to assign the same label to different instances of the same person in different cameras Consistent labeling can be exploited not only for people tracking on wide scenes, but also for posterity logging; multiple views of the same person exploited to improve retrieval for post-analysis HECOL (Homography and Epipolar-based Consistent Labeling): pure geometrical approach [3]

15. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 15 The HECOL system

16. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 The HECOL system (2) Off-line process automatically computes ground- plane homography and epipolar constraints correct correspondence On-line process employs Bayesian-competitive approach with warping of vertical axis and a two- contributions check a2ja2j a1ja1j homography epipolar geometry FG MAP label assignment:

17. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 17 Agenda Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling Probabilistic people trajectory classification Conclusions Motivations The IMAGELAB distributed video surveillance system Motion detection and tracking Multi-camera consistent labeling Probabilistic people trajectory classification

18. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Trajectory acquisition

19. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Path modeling Each trajectory is encoded as a sequence of directions: Each trajectory is modeled as a Von Mises distribution where I 0 is modified zero-order Bessel function of the first kind. The Von Mises parameters are learnt through ML estimator:

20. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Learning phase A training set of trajectories has been collected: 88 trajectories to train the classifier and model the concept of normality Aiming at clustering similar trajectories, not similar directions Parameters’ space clustered with k-medoids, because it is not Euclidean; specific distance metric based on Bhattacharyya distance Analytical and closed-form expression derived for two Von Mises distributions:

21. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Clustering trajectories

22. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Clustering trajectories (2) The medoids constitute the model of normal behavior and are used to build a multimodal mixture distribution of “normality”: –The final distribution is a mixture of Von Mises distribution –The components are the Von Mises pdf of each medoid. –The weights of each component is proportional to the number of training trajectories that fall into a specific cluster –The influence of abnormal trajectories acquired during training is smoothed by clustering and mixture component weighting coefficients

23. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Testing phase Classification of a new path T j is performed by a two- steps approach that: –first selects the best candidate model among the available medoids –and subsequently tests its fitness with the observed data First stepSecond step

24. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Testing phase (2) Model selection exploiting MAP framework to maximize the model’s parameters posterior over the observed path directions (a discrete hidden 1-of-K variable Z is introduced) After the evaluation, the desired model parameters are selected according to the Z i value that maximizes the posterior:

25. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Testing phase (3) After the selection of the model, the path is verified against the model using a first-order Bayesian network : Right side decoupled in two contributions, the first coming by the fitness of the variable against the selected model and the second coming from the range of variability with respect to the previous observed value:

26. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Results for path classification Testing: more than two hours of logging, two sets (of 121 and 135 trajectories) Ground truths based on judges of experts. The experts divided the trajectories into 95 abnormal and 161 normal. The classification rate is 100% for abnormal and 97.5% for normal. It is important to observe that the system correctly detects all the abnormal trajectories generating only false warnings in the case of normal behavior erroneously classified.

27. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Conclusions Complete system for distributed video surveillance presented Single steps commented and results shown Details on high-level trajectory classification for abnormal path detection The overall system has been deeply tested on campus-based controlled scenario Preliminary setup on park (uncontrolled) scenario shows promising results

28. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 Thank you!! 28

29. http://imagelab.ing.unimore.it ICDSC 2007 – Vienna, Austria – 25-28 Sept. 2007 29