1. Prof. Ebroul Izquierdo Head of Multimedia & Vision Group Queen Mary University of London Object Classification based on Behaviour Patterns Virginia Fernandez Arguedas

2. Slide 2 Introduction Technological developments Concern about security Enormous amount of data is generated everyday Lack of resources – human supervision Critical need of automatic object and event classification schemes Mitigate the dependency of constant supervision Widespread of CCTV recording 24/7

3. Object Classification Framework Slide 3

4. Slide 4 Appearance vs behaviour Appearance – based classifiers Visual features are the main source of information Colour, texture or edges Affected by external factors typical in surveillance videos Changes in illumination, occlusions or resolution Behaviour – based classifiers Psychological studies have shown that human beings can routinely recognise the type of object using motion or behaviour patterns, even in large viewing distances where the scene of observation is affected by either poor visibility conditions or in circumstances where other familiarity cues such as appearance are hard to distinguish Exploit temporal and geometrical characteristics to distinguish between semantic categories Eg. Motion, spatial location or velocity Invariant to external factors and general changes in objects appearance OBJECTIVE 1: Discriminative power of behaviour features for object recognition and classification

5. Binary vs Fuzzy Classifiers Binary classifiers Sharp decisions Do not provide information on the confidence of the classifier Fuzzy Classifiers Appeared due to the need of expressing uncertainty Procure higher degree of freedom in any decision-making process Provide information on the confidence on the classifier A certain values on Type-1 Fuzzy Classifiers A range of values on Type-2 Fuzzy Classifiers Slide 5 OBJECTIVE 2: Behavioural fuzzy classifier which progressively discriminate objects by including different degrees of uncertainty in the classification process

6. Surveillance Media Management Slide 6 Motion Analysis Component Behaviour Feature Extraction Behavioural Fuzzy Classifier

7. Slide 7 Motion Analysis Redundant information removal Background quasi constant Adaptive background subtraction Modelling the background as a mixture of Gaussians and each pixel as a weighted mixture of Gaussians Spatial segmentation of the foreground objects Two-pass connected components based on a 8-connection Object detection Temporal segmentation Establishing the correspondence of the spatially segmented objects between frames Linearly predictive multiple hypothesis tracking algorithm based on a set of Kalman filters

8. Slide 8 Motion Analysis - Challenges

9. Behaviour-based Object Representation Stream of spatio-temporal information corresponding to the evolving trajectory of the detected moving objects Due to the evolutionary nature of an object trajectory and the behaviour features trajectory-dependancy, the trajectories provided by the MAC are processed to reduce the amount of artifacts and innacuracies A trajectory is composed by a series of consecutive small direction variations Due to the trajectory-dependant nature of the behaviour features Trajectories are subdivided into tracklets Each tracklet has different orientation Composed-trajectories division algorithm Searches orientation devitions in the trajectories  Breakpoints Temporal and spatial breakpoints Minimising the number of tracklets vs producing the most accurate represantation Slide 9

10. Temporal and Spatial Breakpoints Temporal breakpoints: When an object after a long period of inactivity restarts its movement Highly likeliness to change its trajectory  change in the Behaviour Features Spatial breakpoints: Strong diversion in an object trajectory angle Geometrical procedure  trajectory angle Short-term and long-term Short –term Fast diversion in the trajectory i.e. Sharp angles Long-term: Cumulative sum of direction variations along time Slide 10

11. Each object’s set of blobs Maintain similar physical properties Spatio-temporal information evolve on time Exhaustive analysis of surveillance urban scenarios to determine the most suitable object behaviour attributes Small intra-object variance High inter-object variance Feature selection based on an extrapolation from realistic scenarios from surveillance datasets Geometrical attributes: Trajectory Shape pattern Velocity Each behaviour feature is computed in a projected 2-dimensional space Neglecting the depth in the video perceived by human beings All the behaviour features are trajectory-dependant Slide 11 Behaviour-based Object Representation (II)

12. Behaviour Features (I) Trajectory Main geometrical feature Foundation of the Behaviour Fuzzy Classififer (BFC) Behaviour features trajectory-dependence. An object trajectory is composed by a set of cumulative trajectory variations Each tracklet provides a large amount of spatio-temporal information Measurements: Trajectory angle:α  Basis for the extraction of more compact trajectory measures Global trajectory angle Quadrant Vertical/horizontal object directionality Slide 12

13. Behaviour Features (II) Shape Pattern Object dimensions evolution Size Ratio Object general size Determined by the number of pixels composing its bounding box Shape Ratio Shape proportion Pictured as the bounding box dimensions ratio Velocity High ability to distinguish among semantic categories in certain situations Real-world physical constraints placed on the semantic concept Person favours the classification of the semantic concept Vehicle whenever the object’s velocity overpasses realistic thresholds Ranges of values where the semantic categories are not so easily distinguishable Roads with speed limitation, roads affected by congestion, etc Used to discard in case of ambiguity between semantic concepts Geometrical method on the 2D projected space Slide 13

14. Behaviour Features (III) Velocity: Computation based on The visual distance between consecutive blobs on the 2D-projected space Temporal distance between blobs Slide 14

15. Behavioural Fuzzy Classifier (BFC) Need of a high level of adaptability in the classification stage Fuzzy nature of the behaviour features Fuzzy systems convert human language rules into their mathematical equivalents Procuring an adaptive model able to represent real world BFC consists on a hierarchical rule- based fuzzy classifier built on cascade BFC is achieved into two levels Each individual behaviour feature describes a particular attribute of a moving object with unique requirements and characteristics Slide 15

16. Behavioural Fuzzy Classifier (II) First level Classification of each moving object according to every individual behaviour feature Level built with a set of nested rule-based fuzzy classifier The membership functions applied for each behaviour feature are extrapolated from the marginal training samples and created by analysts from the manually annotated dataset Second level Considers the membership degrees/intervals and individual decisions undertaken by each individual fuzzy classifier in the first level Combine them and extract a final decision based on the objects behaviour rather than their appearance Slide 16

17. Behavioural Fuzzy Classifier (Types) Two types of Fuzzy Logics: Type-1 and Type-2 Fuzzy Logics Study the benefits and impact of the inclusion of different degrees of uncertainty in the classification process Slide 17 Type-1 Fuzzy Logics (1 level of uncertainty) Type-2 Fuzzy Logics (2 levels of uncertainty)

18. Experimental Results Dataset Ground Truth Binary Results Binary vs Fuzzy Classifiers Type-1 vs Type-2 Behavioural Fuzzy Classifiers Appearance vs Behaviour Features Slide 18

19. iLIDS Dataset UK Home Office Parking Vehicle Dataset Outdoor videos 13400 images Variable lighting conditions Dataset Slide 19

20. Evaluation dataset Concepts: Vehicle, Person Ground truth: 1576 objects (manual annotation) Inter-annotated agreement results: 50% Vehicle 6% Person The remainder were annotated as Unknown due to the lack of agreement in the inter-annotation Small blob size Partial occlusion of the object over a 50% Multiple objects coexisting in a blob Slide 20

21. Binary Results Different degrees of uncertainty were considered Two implementations of the BFC: T1BFC and T2BFC Fuzzy classifiers provide a membership label and a membership degree/interval for each detected moving object The membership degree/interval exhibits the reliability on the membership label Classification results based uniquely on the membership label Slide 21 BFC Type Semantic Class TP (%)TN (%)FP (%)FN (%)Accuracy (%) T1BFCVehicle14.8747.112.3225.7161.96 T1BFCPerson2.8149.9047.100.1952.71 T2BFCVehicle32.2318.6340.778.3649.88 T2BFCPerson1.7288.458.551.2790.17

22. T1FL vs T2FL Slide 22

23. Binary vs Fuzzy Classifiers Slide 23

24. Behaviour vs Appearance Slide 24

25. Conclusions SMM provides automatic object classification in urban environments Analysis of behaviour features significant for human being to imitate human beings’ inference procedure Set of novel geometrical extraction algorithms is presented Provide robust behaviour models enabling the classification of objects when their appearance is not clear The hierarchical structure of the fuzzy classifier shows robustness against behaviour outliers and the consideration of the uncertainty of the classifier provides significant improvement against binary classifiers The usage of Type-1 FL for object classification provides a high adaptation to the problem, outperforming the appearance-based object classification The insertion of a higher level of uncertainty, Type-2 FL, within the individual classifiers demonstrated to be a better solution Slide 25

26. Future Work Broadening the surveillance video dataset Including more sophisticated scenarios and more severe external factors (night time) Enlarge the surveillance taxonomy To enable the detection of a larger number of semantic classes Towards the construction of an event detection technique for the recognition of suspicious behaviours in urban environments Slide 26

27. Prof. Ebroul Izquierdo Head of Multimedia & Vision Group Queen Mary University of London Thank you for your attention Questions? Virginia Fernandez Arguedas Virginia.fernandez@eecs.qmul.ac.uk

28. BFC Advantages Rule-based fuzzy classifiers were applied to enable the mathematical modeling of human being’s inference procedure A set of rules understandable for human beings and based on the behaviour patterns were proposed using fuzzy logics Favouring the computation of their mathematical model The application of fuzzy logics into the BFC tacked adaptability The classification through a nested structure provided robustness against outliers as well as a scalable algorithm able to enlarge the number of behaviour features and semantic classes under study For the analysis of more sophisticated scenarios Slide 28