1. Havva Alizadeh Ferdowsi University of Mashhad, WTLab Spring 2011 1

2. Outline Introduction Definition of information fusion Classifying information fusion Architectures and models Ontology + fusion Semantic JDL Future works 2

3. introduction In a nutshell, information fusion can be defined as the combination of multiple sources to obtain improved information. information fusion is used in different application: Robotics Military Intrusion detection WSN (Wireless Sensor Network) DoS (Denial of Service) 3

4. Definition of information fusion Hall and Linas : “the combination of data from multiplesensors, and related information provided by associated databases, to achieve improved accuracy and more specific inferences than could be achieved by the use of a single sensor alone.” Joint Directors of Laboratories (JDL): “multilevel, multifaceted process dealing with the automatic detection, association, correlation, estimation, and combination of data and information from multiple sources.” 4

5. Classifying information fusion Information fusion can be categorized based on several aspects : Classification Based on Relationship Among the Sources(e.g., cooperative, redundant, and complementary data). the abstraction level of the manipulated data during the fusion process (measurement, signal, feature, decision) based on input and output of a fusion process 5

6. Classification Based on Relationship Among the Sources 6

7. Complementary : When information provided by the sources represents different portions of a broader scene, information fusion can be applied to obtain a piece of information that is more complete. Redundant. If two or more independent sources provide the same piece of information, these pieces can be fused to increase the associated confidence. ( to obtain more accurate information) Cooperative. Two independent sources are cooperative when the information provided by them is fused into new information (usually more complex than the original data) that, from the application perspective, better represents the reality. 7

8. Classification Based on Levels of Abstraction According to the abstraction level of the manipulated data, information fusion can be classified: Low-Level Fusion. Also referred to as signal (measurement) level fusion. Raw data are provided as inputs, combined into new piece of data that is more accurate (reduced noise) than the individual inputs. (filters) Medium-Level Fusion. Attributes or features of an entity (e.g., shape, texture, position) are fused to obtain a feature map that may be used for other tasks (detection of an object). This type of fusion is also known as feature/attribute level fusion. High-Level Fusion. Also known as symbol or decision level fusion. It takes decisions or symbolic representations as input and combines them to obtain a more confident and/or a global decision. An example of high- level fusion is the Bayesian approach. 8

9. Classification Based on Input and Output Data In–Data Out (DAI-DAO). deals with raw data and the result is also raw data, possibly more accurate or reliable. Data In–Feature Out (DAI-FEO). uses raw data from sources to extract features or attributes that describe an entity. Here, “entity” means any object, situation, or world abstraction. Feature In–Feature Out (FEI-FEO). works on a set of features to improve/refine a feature, or extract new ones. Feature In–Decision Out (FEI-DEO). takes a set of features of an entity generating a symbolic representation or a decision. Decision In–Decision Out (DEI-DEO). Decisions can be fused in order to obtain new decisions or give emphasis on previous ones. 9

10. Architectures and models Information-Based Models JDL DFD Activity-Based Models Boyd Control Loop Intelligence Cycle Omnibus model Role-Based Models Object Oriented Model Frankel-Bedworth Architecture 10

11. JDL Model (Information based models) 11

12. JDL Model (Information based models) JDL is a popular model in the fusion research community. It was originally proposed by the Joint Directors of Laboratories (JDL) and the Department of Defense (DoD). Sources. providing the input information, and can be sensors, a priori knowledge, databases, or human input. Human Computer Interaction (HCI). HCI is a mechanism that allows human input, such as commands and queries, and the notification of fusion results through alarms, displays, graphics, and sounds. Commonly (query-based interfaces) 12

13. Level 0 (Source Preprocessing). Various preprocessing of data is needed, e.g. normalization of signal measurements, handling of missing values in the data set, handling of incomplete data sets, filtering out low quality measurements. Level 1 (Object Refinement). Object refinement transforms the data into a consistent structure. attempts to locate and identify objects. Level 2 (Situation Refinement). Situation refinement tries to provide a contextual description of the relationship among objects and observed events. It uses a priori knowledge and environmental information to identify a situation. 13 JDL Model (Information based models)

14. Level 3 (Threat Refinement). evaluates the current situation projecting it into the future to identify possible threats, vulnerabilities, and opportunities for operations. Level 4 (Process Refinement). This is a meta-process, responsible for monitoring the system performance and allocating the sources according to the specified goals. (e.g. QoS, Energy maps in WSN) 14 JDL Model (Information based models)

15. λ- JDL 15

16. 16 Elements of Situation Assessment

17. 17 Elements of Threat Assessment

18. 18 SAW High-Level Ontology

19. Battlefield Relations Ontology 19

20. The Boyd Control Loop or OODA (Observe, Orient, Decide,Act) Loop is a cyclic model composed of four stages: Observe. Information gathering from the available sources. Orient. Gathered information is fused to obtain current situation. Decide. Specify an action plan in response to understanding of situation. Act. The plan is executed. 20 OODA Loop (Activity based models)

21. 21 OODA Loop (Activity based models) Level 1,2,3 JDL Level 0 JDL Level 4 JDL It is n’t in JDL

22. Object-oriented model provided by Kokar et al. [2000] in which four roles are identified: Actor. Responsible for the interaction with the world, collecting information and acting on the environment. Perceiver. Once information is gathered, the perceiver assesses such information providing a contextualized analysis to the director. Director. Based on the analysis provided by the perceiver, the director builds an action plan specifying the system’s goals. Manager. The manager controls the actors to execute the plans formulated by the director. 22 Object-Oriented Model (Role-Based Models)

23. 23 Object-Oriented Model (Role-Based Models)

24. METHODS, TECHNIQUES, AND ALGORITHMS Inference : Bayesian Inference Dempster-Shafer Inference Fuzzy Logic Neural Networks: (also for Automatic Target Recognition (ATR)) Abductive Reasoning Estimation : Maximum Likelihood Maximum A Posteriori (MAP) Least Squares Moving Average Filter Kalman Filter Particle Filter 24 Reliable Abstract Sensors Fault-Tolerant Averaging The Fault-Tolerant Interval Function

25. 25

26. heterogeneity of the data within distributed systems : Structural heterogeneity : means that different information systems store their data in different structures. Semantic heterogeneity : considers the content of an information item and its intended meaning. In order to achieve semantic interoperability in a heterogeneous information system, the meaning of the information that is interchanged has to be understood across the systems. 26 Motivation to ontology

27. Three main causes for semantic heterogeneity: Confounding conflicts : occur when information items seem to have the same meaning, but differ in reality. Scaling conflicts : occur when different reference systems are used to measure a value. Examples are different currencies. Naming conflicts : occur when naming schemes of information differ significantly. (homonyms and synonyms) The use of ontologies for the explication of implicit and hidden knowledge is a possible approach to overcome the problem of semantic heterogeneity. 27 Motivation to ontology

28. Ontologies introduced as ”explicit specification of a conceptualization” Therefore, ontologies can be used in an fusion task to: describe the semantics of the information sources to make the content explicit they can be used for the identification and association of semantically corresponding information concepts. 28 The Role of Ontologies

29. 29 Semantic information fusion system based on JDL

30. Single Ontology approaches: use one global ontology providing a shared vocabulary for the specification of the semantics All information sources are related to one global ontology. Single ontology approaches can be applied to integration problems where all information sources provide nearly the same view on a domain. single ontology approaches are susceptible for changes in the information sources which can affect the conceptualization of the domain represented in the ontology. 30 Three main ontology architectures

31. Multiple Ontologies approaches: Each information source is described by its own ontology Each source ontology can be developed their ontologies easily. The lack of a common vocabulary makes it difficult to compare different source ontologies. To overcome this problem, the inter-ontology mapping is needed. Inter-Ontology Mapping approaches: Defined Mappings Lexical Relations Top-Level Grounding Semantic Correspondences 31 Three main ontology architectures

32. Hybrid Approaches : Similar to multiple ontology approaches the semantics of each source is described by its own ontology. But in order to make the local ontologies comparable to each other they are built from a global shared vocabulary. 32 Three main ontology architectures

33. Provide dataset (military,medical, weather,…) Provide related ontologies Provide Rule database (by simulator and then data mining) Ontology-based object detection Ontology-based Situation awareness Semantic threat refinement Evaluation (correct result, time to answer) 33 Steps of work

34. 34 Thanks a lot ?