O NTOLOGICAL R EPRESENTATION OF C ONTEXT K NOWLEDGE FOR V ISUAL D ATA F USION Juan Gómez Romero Miguel A. Patricio Jesús García José M. Molina Applied Artificial Intelligence Research Group (GIAA) University Carlos III of Madrid

Objective Semantic representation of visual information, both perceived and contextual, to facilitate fusion of hard and soft entries in surveillance applications To formalize the heuristics Sensor-based data vs. Contextual and common-sense knowledge

Outline 1.Context in Visual Data Fusion 2.Architecture & Contents of our Model 3.Conclusions and Future Work

Definition of context  Context is “any information (either implicit or explicit) that can be used to characterize the situation of an entity” [1]  Computer Vision  Additional information about the scene entities [2] Scene environment Parameters of the recording Previously computed information User-provided information (soft entries!) [1] A. Dey, G. Abowd. “Towards a Better Understanding of Context and Context-Awareness,” CHI Workshop on the What, Who, Where, When, and How of Context-Awareness, The Hague, Netherlands, [2] F. Bremond, and M. Thonnat, “A context representation for surveillance systems,” ECCV Workshop on Conceptual Descriptions from Images, Cambridge, UK, 1996.

Necessity of Context Knowledge for High-Level Information Fusion Track 008 pos () vel () Track 010 pos () vel() Tracking L1L2-L3 Person Entry > Entering Mirror > Reflection Column Person 1 is (Entering through Entry 2) and (Reflected by Mirror 1) Interpretation User-Provided Context Representation & Reasoning

Proposal Use of ontologies to represent context knowledge for visual data fusion

Ontologies for Context Management  Ontologies: “Formal, explicit specifications of a shared conceptualization” [3]  An ontology is a knowledge model which describes from a common perspective the objects in a common domain using a language that can be processed automatically.  Based on Description Logics (DLs)  DLs are a family of logics to represent structured knowledge  Inferences can be performed: consistency, subsumption, membership, etc.  Basic constructs: Concepts, Relations, Individuals, Axioms  Standard: The Web Ontology Language (OWL) [3] R. Studer, V. R. Benjamins, & D. Fensel. “Knowledge engineering: principles and methods”. In: Data Knowledge Engineering (1998). Pp. 161–197.

Proposal Use of ontologies to represent context knowledge for visual data fusion  Logic-based representation of fusion information  Associated reasoning procedures  Abstract description of the scenes  Better interpretability and easier interaction with users  Extensible and Reusable:  New elements can be easily added to the model  The model can be reused (particularly, by generalization & specialization) in different domains  Standard languages and tools  Less effort to deal with the models

Contribution Ontology-based model to manage contextual and sensorial data in visual fusion systems

Outline 1.Context in Visual Data Fusion 2.Architecture & Contents of the Model 3.Conclusions and Future Work

JDL-based architecture Ontological Model Descriptive Knowledge (TBox): Definition of concepts, relations, etc. Intensive Knowledge (ABox): Instantiation for a concrete scene

JDL-based architecture: Inputs (I) Hard Inputs: Sensor Data 1. Tracking data obtained by a (classical) tracking algorithm 2. Identification data 3. Non visual sensor data

JDL-based architecture: Inputs (II) Soft Inputs: Human-generated Data 1. Contextual information 2. Context-based rules

JDL-based architecture: Outputs Outputs 1. Situation Assessment 2. Impact Assessment 3. Visualization of the interpreted situation

JDL-based architecture From Data to Information: Abductive Reasoning 1. Tracking: Moving entities 2. Correspondence: Association between possible objects and tracks 3. Recognition: Activity identification 4. Evaluation: Computation of the impact of an activity

JDL-based architecture: TREN ontology L1 – T RACKING E NTITES O NTOLOGY (T REND )  Ontological representation of low-level data from the tracking algorithm: frames, tracks and track properties  Temporal evolution of the tracks: tracks have associated track snapshots  Flexible representation of properties: qualia spaces (DOLCE ontology)

JDL-based architecture: SCOB ontology L1-L ½ -- S CENE O BJECTS D ESCRIPTION O NTOLOGY (S COB )  Objects of the scene: entry, exit, person, column, etc.  Static (contextual) and Dynamic (tracked) objects  Object properties (change in time)

JDL-based architecture: ACTV ontology L2 – A CTIVITY D ESCRIPTION O NTOLOGY (A CTV )  Activities of the scene and connections with the objects involved: grouping activity + grouped objects  Activities taxonomy largely based on: C. Fernández, and J. González, “Ontology for Semantic Integration in a Cognitive Surveillance System,” 2nd Int. Conf. on Semantic and Digital Media Technologies, Genoa, Italy, 2007, pp

JDL-based architecture: IMPC ontology L3 -- I MPACT D ESCRIPTION O NTOLOGY (I MPC )  Abstract description of the impact of activities  Impact concept (reification of the hasImpact relation)  Impact taxonomies or restrictions according to context could be implemented

Overview of the In-Use Ontological Model Specific Model  Specialization of the template concepts provided in the General Knowledge Model  PETS2002 sequence Abductive Rules  Rules with ontological terms to infer information of a higher level from information of a lower level  Example: If the distance between two people is decreasing, then they are grouping

Outline 1.Context in Visual Data Fusion 2.Architecture & Contents of the Model 3.Conclusions and Future Work

Summary  Ontological model for representing contextual and perceived data for visual data fusion  Formal description of scenes and reasoning, from low-level to high-level (intra-level reasoning)  Logic-based mechanisms (rules) to infer high-level information from low-level data (inter-level reasoning)  Extensible to different applications (e.g. surveillance)  Temporal evolution of the scenes

Future work  Full integration with tracking software  Adaptation (simplification) of representation and reasoning when response time is constrained  Incorporation of different data sources, not only visual  Test and validate results in different application areas  Development of ontologies and rule bases  Feedback to the low-level algorithms from the high- level  How tracking errors can be detected (or predicted) and solved when the situation has been identified?

T HANK Y OU ! Q UESTIONS …