Challenges in Information Fusion Technology Capabilities for Modern Intelligence and Security Problems Speaker: Prof. Sten F. Andler Director, Infofusion Research Program University of Skövde, Skövde, Sweden (*) Author: Dr. James Llinas Center for Multisource Information Fusion University at Buffalo, Buffalo, New York, USA (*)

Key Information Fusion Challenges Driven by Operational Problems and Modern IT Heterogeneity of Data, Information Common Referencing and Data Association Impacts Dealing with Semantics The Entry of Graphical Methods Architecting Systems and Analytic Frameworks

Heterogeneity of Data/Information Observational – “Hard” Sensor Data and “Soft” linguistic/reported/unstructured Data Open-source & Social Media – Issues: Mostly in linguistic form; Trust, Volume, Formats, Modalities Contextual differences – Issues: Format, Middleware reqmt, dynamics, relevance Ontological differences – Issues: Multiple-ontology cases, semantics, dynamics, relevance Learned knowledge – Issues: integrating inductive and other inferencing procedures Heterogeneity from modern IT capabilities/problems and networked systems Lack of reliable a priori knowledge to support dynamic deductively-based reasoning  “Weak Knowledge” problems

Soft (linguistic) data Soft (linguistic) data -- New preprocessing Front Ends: requirement for semantically robust Text Extraction/NLP processes – Marginally available today – If not extracted, properly labeled entities never enter the Fusion process – If not tagged with some level of (reliable) uncertainty/confidence, entity uncertainty not considered Confounds both Common Referencing and Data Association Exploiting Contextual Data Exploiting Contextual Data requires Middleware to condition data in a form useable by Fusion process (native form-to-useable form) – Can also require hybrid algorithms, eg context-aided Kalman Filter designs multiple Ontological versions In networked systems, there can be multiple Ontological versions being used – Creates a need for ontological normalization (Common Referencing function) – Also impacts Data Association; inconsistent nomenclature will prevent feasible associations Information learned in real-time Level 4 Knowledge Management Information learned in real-time creates a Level 4 Knowledge Management functional requirement, and real-time adaptation that can include dealing with out-of-sequence evidence (retrospective adaptation) Some Impacts due to Data Heterogeneity

Common Referencing Common Referencing – Temporal alignment within streaming Soft data feeds is challenging Dealing with linguistic tense: past/present/future – Impacts correct Temporal Reasoning » Creates a need for agile Temporal Reasoning – Networked environments open the possibility for inconsistent forms of uncertainty representation Creates a need for uncertainty transforms, normalization methods Data Association Data Association – Major impact due to Soft (linguistic) data and availability of Relational links Association now of higher dimension: Entities/attributes and inter- entity Relations — becomes a Graph Association problem New scoring functions required; eg Relational similarity Some further Impacts regarding Common Referencing and Data Association

Representative Impacts regarding Common Referencing and Data Association, cont. G. Tauer, R. Nagi, M. Sudit, The graph association problem: Mathematical models and a lagrangian heuristic, Naval Research Logistics (NRL) Volume 60, Issue 3, pages 251–268, April 2013

Graphs as a Representational Form Graphs as a Representational Form – The standard for language representation – Deals with Entities and Relations – Quantitatively-based; visually manageable Graph-based Analytics Graph-based Analytics – Framework for Data Association as shown – Evidential searching/matching (supports query-based, discovery-based analysis) Variety of Graph-Matching paradigms, issues – Stochastic due to tagged uncertainties in graph elements – Incremental to handle streaming real-time data – Large scale to handle “Big Data”; eg Cloud-based Representative Impacts regarding Graphical Forms and Operations

strategies for semantic “control” Optimal strategies for semantic “control” – control of semantic complexities – Rigorous control of Ontologies – Controlled vs Uncontrolled Languages Eg Battle Management Language – Robust Text Extraction, NLP – Role of Human Mediators in system architecture Speed (automation) vs semantic accuracy Semantic Uncertainty Semantic Uncertainty Vague predicates; issue of Truth—leads to 3-valued forms of Uncertainty Representation Some further Impacts regarding Semantics

extent of reliable a priori dynamic knowledge about the domain is limited Many problems are “Weak Knowledge” problems wherein the extent of reliable a priori dynamic knowledge about the domain is limited combine deductive and inductive (or abductive) This motivates an approach that must combine deductive and inductive (or abductive) methods in an effective way – These tend to require technologies that support discovery and learning-based hypothesis-formulation strategies new inferencing methods Methods such as Complex Event Processing, Probabilistic Argumentation, Graph-based Relational Learning are some of the new inferencing methods being studied. Some Impacts regarding System Architectures and Analytical Frameworks

* Integrating the Data Fusion and Data Mining Processes Ed Waltz, Natl Symp on Sensor and Data Fusion, 2004 Earliest Thoughts on Combining Inductive and Deductive Inferencing for Fusion* Representative Architectures: Inductive + Deductive

Representative Architectures: Hard and Soft Fusion Processes; Disparate Analytic Tools encin g n g on - g Hard (sensor) fusion Enterprise Service Bus Core Enterprise Servces Information (Evidence) Services (Sensor) Data and Computational Services Evidence and Entity -estimate Foraging Services Sensemaking Services IntelCell–or–CompanyOpnsIntellSupport Team Analytic Support Services Soft (intel) fusion

Summary Requirements for Data and Information Fusion Processes and Systems have gone far beyond the goal of estimating properties and geometries of entities – Dealing with complex Semantics, inter-entity Relations, Social Media and other Contextual effects, complex Temporal dynamics, and Heterogeneous Data have made the design of IF Systems a markedly new challenge. Incremental advances and accomplishments are being realized but there is much to be done Major advances are needed in dealing with more complex inferencing challenges to support efficient learning and discovery processes. New partnerships are needed across various multidisciplinary areas in order to address these new complexities