1. VISION for Security Monique THONNAT ORION INRIA Sophia Antipolis

2. 18/03/05 CS M. Thonnat 2 Which Security Problems? Safety and security of goods and human beings How? Data captured by video surveillance cameras Original video understanding approach mixing: computer vision: 4D analysis (3D + temporal analysis) artificial intelligence: a priori knowledge (scenario, environment) software engineering: reusable VSIP platform Introduction

3. 18/03/05 CS M. Thonnat 3 Definition: real time and automated analysis of video sequences video understanding= from people detection and tracking to behavior recognition Recognition of complex behaviors: of individuals (fraud, graffiti, vandalism, bank attack) of small groups (fighting) of crowds (overcrowding) interactions of people and vehicles (aircraft refueling) Video Understanding for Security

4. 18/03/05 CS M. Thonnat 4 Video Understanding 4 D analysis: multi-cameras tracking Video understanding People detection and tracking Scenario recognition A PRIORI KNOWLEDGE: 3d models of the environment Camera calibration Scenario Models Alarms People detection and tracking Interpretation of the videos from pixels to alarms

5. 18/03/05 CS M. Thonnat 5 Impact: Visual surveillance of metro stations, bank agencies, trains, buildings and airports 5 European projects (PASSWORDS, AVS-PV, AVS-RTPW, ADVISOR, AVITRACK) 4 contracts with End-users companies (metro, bank, trains) 2 transfer activities with Bull (Paris) and Vigitec (Brussels) Cooperation over more than 11 years with partners Creation of a start-up (spring 2005) Video Understanding

6. 18/03/05 CS M. Thonnat 6 Typical problems Metro station surveillance Surveillance inside trains Building access control Airport monitoring

7. 18/03/05 CS M. Thonnat 7 Behavior recognition: approach based on a priori knowledge model of the empty scene (3D geometry and semantics) models of predefined scenarios a language for representing scenarios based on combination of states and events more than 20 states and 20 events can be used a reasoning mechanism for real time detection of states, events and scenarios (e.g. temporal reasoning, constraints solving techniques) Video Understanding

8. 18/03/05 CS M. Thonnat 8 Video Understanding: 3D Scene Model 3d Model of 2 bank agencies objet du contexte mur et porte zone d’accès salle du coffre rue salle automates zone d’entrée de l’agence zone des distributeurs zone de jour/nuit zone devant le guichet zone derrière le guichet zone d’accès au bureau du directeur zone de jour porte d’entrée porte salle automates armoire guichet commode Les Hauts de Lagny Villeparisis

9. 18/03/05 CS M. Thonnat 9 States, Events and Scenarios : State: a spatio-temporal property involving one or several actors on a time interval Ex : « close», « walking», « seated» Event: a significant change of states Ex : « enters», « stands up», « leaves » Scenario: a long term symbolic application dependent activity Ex : « fighting», « vandalism» Video Understanding

10. 18/03/05 CS M. Thonnat 10 Results for Bank Monitoring Bank attack scenario description : scenario Bank_attack_one_robber_one_employee physical_objects: ((employee : Person), (robber : Person), z1: Back_Counter, z2: Entrance_Zone, z3: Front_Counter, z4: Safe, d: Safe_door) components: (State c1 : Inside_zone(employee, z1)) (Event c2 : Changes_zone(robber, z2,z3)) (State c3 : Inside_zone(employee, z4)) (State c4 : Inside_zone(robber, z4))) constraints : ((c2 during c1) (c2 before c3) (c1 before c3) (c2 before c4) (c4 during c3) (d is open))

11. 18/03/05 CS M. Thonnat 11 Video Understanding for bank surveillance

12. Examples : Brussels and Barcelona Metros Exit zone Jumping over barrier Blocking Overcrowding Fighting Group behavior Crowd behavior Individual behavior Group behavior Results in Metro Surveillance 12

13. 18/03/05 CS M. Thonnat 13 Video Understanding: Conclusion Hypotheses: fixed cameras 3D model of the empty scene predefined behavior models Results: + Behavior understanding for Individuals, Groups of people, Crowd or Vehicles + an operational language for video understanding (more than 20 states and events) + a real-time platform (5 to 25 frames/s)

14. 18/03/05 CS M. Thonnat 14 Knowledge Acquisition Design of ontology driven knowledge acquisition: video event ontology (T. Van Vu PhD) Design of learning techniques to complement a priori knowledge: visual concept learning(Nicolas Maillot PhD) scenario model learning (A. Toshev) Reusability is still an issue for vision programs Use of program supervision techniques: dynamic configuration of programs and parameters (B Georis PhD) Video event detection Finer human shape description : 3D posture models (B. Boulay PhD) Video analysis robustness: Uncertainty management (M. Zuniga PhD) Conclusion: Where we go

15. 18/03/05 CS M. Thonnat 15 Computer Vision Mobile object detection (Wei Yun I2R Singapore) Tracking of people using geometric approaches (T. Ellis et al. Kingston University UK) Event Recognition Probalistic approaches HMM, DBN (A Bobick Georgia Tech USA, H Buxton Univ Sussex UK) Reusable platform Realtime video surveillance platform (Multitel, Be) State of the Art