Visualization and Fusion Craig Scott, Aisha Page, Olusanya Soyannwo, Hamzat Kassim Paul Blackmon, Pierre Knight, Francis Dada Engineering Visualization Laboratory

Outline Overview Image/Spatial Fusion Motion detection and object tracking Synthetic Battlespace Conclusion

Overview The objective of this research is to advance ideas that apply to how to enhance battlespace awareness and move tactical decision making closer to the field soldier by applying image and spatial data fusion concepts.

BORG Experimental Architecture & Platform IMPACT / PASTA / P2P - Willy & Damian (VS, Hendler - UMD) Java/C++ Sequal Server/Oracle UML Cognitive Models Sun JXTA B3A Imagery DB (Roy George Clark-Atlanta) Soldier Highlights Craig JCDB GIS...Requirements, Cognitive Deficiencies Pamela,Leroy Autonomous Fusion and Navigation Distinct Sources Data Streams Content Based Messaging System Willy & Damian (Andrew Cowley PNNL) RSS RSS... Advanced Interactions Story (VS - UMD) Visualizations Craig Commander/ Analysts Tactical Operations Center Workstations Command and Control Interactive Fusion and Navigation Summary of Source Website (Hendler - UMD), Vignette, Requirements Leroy, Pamela Plans (Monmouth) WWW Heterogeneous Data sources Knowledge Engineering, Cognitive Models, HCI Leroy,Pamela Seed Demo C++ DirectShow Windows AVI video

Issues/Approach Exploiting Level of Detail Perception –Multiple channel architectures –Area-of of-interest filters –Subscription-based aggregation Exploiting Temporal Perception –Render the entity in an accurate location as long as the local user does not interact, Distingish active and passive entities

Image/Spatial Data Fusion Image/Spatial Data Fusion: combining complete spatially filled sets if data in 2-D or 3D Registration Combination Reasoning Viewing Volume or Plane Terrain Layer Meaning Data Structures 2D optical photos & video, FLIR, SAR 3D terrain data, buildings, vehicles, weather models, LADAR

Tracking Studies Objective –To be able to provide one solution to soldiers query about unrecognizable objects (tanks, airplanes, people etc.) through object tracking Problem –Finding the best tracking algorithm efficient enough to display and track desired object –Implementing different algorithms into frame work which will display best algorithm suitable for the current field operation Solution –Implement various tracking algorithms with varying environmental conditions –Validate effectiveness through perceptual studies

Motion and Object Tracking Algorithms Background subtraction Motion templates Optical flow Active contours Estimators

Background Subtraction (contd) μ t = αy+ ( 1 –α)μ t –μ - updated image –t – time constant –α – learning rate constant Specifies how fast (responsive) background model is adapted to changes 0 <= α <= 1 –y – new observation at time t

Active Contour Active Contours –Curves defined within an image domain –Can move under influence of Internal forces coming from within the curve itself External forces computed from the image data Allows the computer to generate curves that move within images to –locate object boundaries –find other desired features within an image

Active Contour (contd) Energy equation associated with snake E= E int +E ext –E int - internal energy formed by the snake configuration E int = E cont + E curv –E cont – Contour continuity energy »Minimizing E cont over all the snake points, causes the snake points to become more equidistant –E curv – Contour curvature energy »The smoother the contour is, the less the curvature energy

Active Contour (contd) –E ext is the external energy formed by external forces affecting the snake E ext = E img + E con –E img – Image energy –Two variants of image energy are proposed: »E img = -I, where I is image intensity »E img = -||grad(I)||, Snake is attracted to image edges –E con – Energy of additional constraints

KLT: Kanade-Lucas-Tomasi Feature Tracker Problem –Complex changes occur between frames Good features are located by: –Examining the minimum eigenvalue of each 2x2 gradient matrix Key components to feature tracker –Accuracy: relates to local sub-pixel accuracy attached to tracking –Robustness: relates to sensitivity of tracking with respect to changes of: Lighting Size of image motion Goal –To find location on second image –Such that, image one and two are similar

Computing Image Motion Residual function: J – second image I – first image W – given feature X – point within image w(x) – weighting function A = 1 + D d – translation (uniform movement) of feature windows center; from one frame to another D – deformation (change of shape) matrix –Used to determine if the first and current frames match

Feature Selection Means to select point in image I Selection maximizes quality of tracking Central step of tracking – computation of the optical flow –Optical flow: motion of brightness patterns in image –At this critical step the minimum eigenvalue must be larger than a threshold This characterizes pixels that are easy to track

Test Scene Virtual Laboratory for Data Fusion Studies

Tracking Example Background Subtraction

Battlespace Visualization Battlespace visualization is the process whereby the commander –Develops a clear understanding of the current state with relation to the environment. –Envisions a desired end state that represents mission accomplishment. –Visualizes the sequence of activity that moves the commanders force from its current state to the end state. Army Geospatial Guide for Commanders and Planners, TC 5-230, November 2003.

Synthetic Battlespace Research Issues Perception/Trust Agent Roles & Interaction 3D Scene Reconstruction Automated Feature Extraction Registration Accuracy LOD/Bandwidth Tradeoff –As DoD looks to the future, increasing demands on the warfighter dictate the increased use of simulations in operational situations. Ideally, the simulation power is placed at the immediate disposal of the warfighter so that it can be accessed and employed when needed.

Conclusion We have implemented rudimentary prototype object tracking algorithms within a seed demo application to illustrate image/spatial fusion concepts. The synthetic battlespace concepts require integrating present joint simulation technology with fusion research concepts to articulate a COP that is easy to understand and react to. The commercial game market supplies a significant amount of talent and resource$ (market) to fuel this area