Command Post of the Future Limited Objective Experiment-1 Presented to: Information Superiority Workshop II: Focus on Metrics March 2000 Presented by: Dr. Richard E. Hayes Evidence Based Research, Inc Spring Hill Road, Suite 250 Vienna, VA

2 Agenda HEAT and Quality of Information CPOF and Quality of Awareness Summary

3 Headquarters Effectiveness Assessment Tool (HEAT) HEAT is a set of consistent procedures that measure the effectiveness of a military headquarters or command center Quantitative, objective, and reproducible assessment of quality of C2 processes and overall effectiveness of directives Applied more than 50 times to exercises and real world applications in more than 200 command centers HEAT views headquarters as analogous to an adaptive control system that influences environment through plans (directives) Headquarters can be judged by viability of plans –Remain in effect for intended period –Include contingencies Diagnostic scores for quality of C2 processes

4 HEADQUARTERS DEVELOP ALTERNATIVE ACTIONS UNDERSTAND MONITOR PREDICT CONSEQUENCES DECIDE DIRECT QUERY INFORM ENVIRONMENT: - OWN & ENEMY FORCES - PHYSICAL - POLITICAL & ECONOMIC Awareness HEAT Analytic Structure

5 HEAT Information Measures Completeness Are all the required elements present? Correctness Does the information match the truth? Consistency Is the information the same across all nodes? Precision Was the information presented at the proper level of precision? Currency (Latency) How old is the information?

6 The goal of CPOF is to shorten the commander’s decision cycle to stay ahead of the adversary’s ability to react. Command Post of the Future Command Post of the Future (CPOF) is a DARPA program that aims to: Increase Speed and Quality of Command Decisions –Faster recognition and better understanding of changing battlefield situation –Faster and more complete exploration of available courses of action Provide More Effective Dissemination of Commands –COA capture for dissemination of commander’s intent –Status and capability feedback from deployed operators Enable Smaller, More Mobile and Agile Command Structures –More mobile, distributed command element –Smaller support tail & reduced deployment requirements

7 Limited Objective Experiment - 1 (LOE -1) Hypothesis: Tailored visualizations will improve Situation Awareness MOE: Correctness of Situation Awareness comprehension Quality of Pattern Recognition Level of Entity Retention

8 Force-on-Force Insurgency Situation 13 Situation 10 Situation 5 Situation 4 Less Complex More Complex LOE-1 Scenario Space

9 Data Collection 157 valid runs over 5 days (98% of design goal of 160) Data Collection Vehicle - Debrief Questionnaire Questions divided into four major elements: overall Situation Assessment, Patterns, Elements, and a Sketch of the situation Military experience data collected for each subject Run randomization compensated for variations between coders and subjects

10 Top Level Measures Overall Situation Awareness –Picture of the battlespace - enemy posture and intentions, friendly possibilities for favorable action Pattern Recognition –Dynamics of the battlespace - relative positions, strength, and capabilities Element Retention –Locations, status, events over time Ability to draw a general Sketch with relative positioning of important entities in space and time

11 1.You have three minutes to brief your commander on your assessment of the situation, what do you tell him? 1a. In this situation what are the friendly: Opportunities? Risks? 1b What are the adversary’s: Offensive capabilities? Defensive capabilities? Vulnerabilities? Intentions? 1c. What are the vulnerabilities for own forces? 2a. If you are not certain about adversary intentions and capabilities, what are the different possibilities? 2b. If you are not certain, what information would you require to resolve this uncertainty? 3a. What specific elements or element combinations lead you to your situation assessment? 3b. Please make a sketch of the situation, including the specific elements or element combinations cited in your previous answer. 3c. Was any information you would find valuable missing from the presentation of the situation? 4. What was good and or bad about this display? What would you like to see added, removed or changed? Debrief Questionnaire

12 Metrics Overall Prompted% of key componentswrong inferences added inferences Unprompted% of key componentswrong inferences added inferences Patterns % of pre-identified wrong added Elements % of important elements wrong added Sketch % of pre-identified elements wrong added temporal pattern geographic pattern

13 Interpretation CPOF Technologies Generated: –Better Situation Awareness (Higher Mean or X) –CPOF Technologies Performance Improves for Prompted CPOF Technologies Significantly Outperform Control in Overall Scores Prompted N=157 p=.007 Unprompted N=157 p=.058 s x Control CPOF Technologies

14 CPOF Technologies Significantly Outperform Control in Complex Situations Interpretation: CPOF Technologies generated better Situation Awareness (higher mean or x) in complex situations Unprompted Prompted N=78 p= s Control CPOF Technologies x

Treatment B Outperforms Treatment A in Overall Situation Awareness in Situation 13 Unprompted N=29 p=.073 Prompted N=29 p=.002 s x Treatment B Treatment A Interpretation: Treatment A used icon visualization scheme that subjects stated was confusing

16 Today’s Technology Allegiance Color Coding Treatment B Blobs and Allegiance Color Coding Treatment A Function Color Coding The Force-on-Force Visualization Treatments

17 Summary Information does not equal awareness Natural layers of overall patterns and elements Measures must always be developed in a military context Overall Patterns Elements Incorrect Richness