1. Fusion and Automation Fusion and Automation Human Cognitive and Visualization Issues Jean-Remi Duquet Marc Gregoire Maura Lohrenz Kesh Kesavados Elisa Shahbazian Tore Smestad Amy Vanderbilt Margaret Varga

2. What is Data (& Information) Fusion? The most recent definitions of the JDL DF levels are listed below: Level 0:Sub-Object Data Association and Estimation (pixel/signal level data association and characterization) Level 1:Object Refinement (object continuous state (e.g., kinematics) estimation, discrete state (e.g., object attribute type and identity) estimation) Level 2:Situation Refinement (object clustering and aggregation, relational analysis, communications and contextual estimation from multiple perspective) Level 3:Significance Estimation (situation implication, event prediction, consequence prediction, opportunities and vulnerability assessment) Level 4:Process Refinement (adaptive processing through performance evaluation and decision/resource/mission management). From Steinberg, Bowman and White (1999) Revisions to the JDL DF Model, SPIE vol. 3719, pp. 430-441.

3. Assumptions  Assume there is a DF system that satisfies our requirements  Rapid prototyping of this DF system could be achieved with the following: 1.A NATO agreed upon set of modular ontologies and an environment for allies to augment these 2.NATO guidelines to assess the requirement for DF capabilities and DF automation 3.NATO guidelines for selecting DF architecture and interfaces, given the system requirements 4.A mechanism to propose and develop a rule set to guide the selection of choices in the DF design 5.A NATO repository of data/information fusion capabilities and algorithms 6.A tool that guides the developer to put together the DF system based on the input information (wizard)

4. Visualization Challenges 1.Conveying the measure of uncertainty to the Operator, with minimal clutter 2.Level of Automation a.How to determine when to apply a DF decision automatically and when to provide the capability as a decision support tool for the Operator b.Context-based automation c.User-selected automation d.Level of awareness management by exception e.Self-monitoring of DF 3.How should the existence of multiple hypotheses be conveyed to the Operator?

5. Challenges (continued) 3.How should the pedigree information be conveyed and what decision support is necessary for the interpretation of the pedigree?  Visualization of the credibility of the source 4.Level of Interaction  How should the Operator participate/aid/veto in the DF decisions? 5.How to visualise and address conflict between DF decision sensor and/or Operator decision? 6.Measuring system effectiveness  Maximise the information contents

6. Design Methodology for the Visualization System  Proposed design based on design processed developed by Buffalo Ontology workshop held in March, 2002 …

7. World Information Requirements to support Decision Making Ontology Abstract Relationships State Change Comparison Similarity Absolute Magnitude Display Instantiations Known Methods for Knowledge Elicitation and Requirements Gathering Mutually Informing Problem Context Physical Environment Experience Culture Hardware Data Availability Integration with other required forms Constrains display concept selection Empirical Evaluation Suite of Representational Elements Many to Many Mapping From Kesavadas and Llinas, et al. (2002) Ontology Workshop, SUNY Buffalo

8. World Sensors Fusion Processes Display Person From Kesavadas and Llinas, et al. (2002) Ontology Workshop, SUNY Buffalo

10. Recommendations  There are many visualization tools to represent risk, uncertainty, and rate of change, but no good guidelines for how best to implement these tools optimally (100%) or sub- optimally (70%)!  There is a need for an ontology that accounts for specific fusion and automation needs (e.g., uncertainty).  Need both subjective and objective evaluation criteria.  Would help to have a standard (NATO?) testbed for prototyping purposes.  Recommend a NATO archive of available tools, guidelines, etc. to help rapid prototyping, analysis, and building data fusion systems.  Need an integrated process model that flows smoothly through all stages of data fusion / visualization design.