1. Analyses of Selective Military Sensemaking Cases Prepared for: CCRP Sensemaking Symposium 23-25 October 2001 Prepared by: Evidence Based Research, Inc. Richard E. Hayes, Ph.D. Stevana M. Allman John R. M. McDaniel Cara N. Christie Telyvin F. Murphy Eric J. Cochrane Kristina M. Thompson Stacey R. LakindWilliam S. Wood

2. Purpose Exploratory empirical analysis of military Sensemaking in order to shed light on: –which parts of the process tend to go well –which parts of the process tend to go badly Examination of the military Sensemaking process to identify elements associated with success and failure

3. Information Domain Cognitive Domain Physical Domain Objects/events Data (representation) Information (data in context) Decision processes Directives Requests for support Queries Reports Efforts to consult Actions Planning Missions Assets Boundaries Schedules Contingencies Synchronization Judgment Prior knowledge Mental Models Emotions Physiological Factors Beliefs Perceptions “Deep” understanding of situation Cause and effect Temporal relations Dynamic futures Opportunities & Risks S h a r e d Sensemaking Values Anticipated dynamic futures Alternatives perceived Command Intent Choices among alternatives including contingent choices Choices to wait Choices to seek information Choices to consult others S h a r e d C o n s t r a i n t s Red Blue S h a r e d Capabilities & Other Intentions Time & Space M i s s & i o n E n v i r o n m e n t Uncertainty Awareness Decision Support Models & Tools Sensemaking: Conceptual Framework

4. Data (representation) Information (data in context) Information Domain Cognitive Domain Physical Domain Objects/events Decision processes Decisions Choices among alternatives including contingent choices Choices to wait Choices to seek information Choices to consult others Directives Requests for support Queries Reports Efforts to consult Actions Planning Missions Assets Boundaries Schedules Contingencies Synchronization Judgment Red Blue Othe r Capabilities & Intentions Time & Space Opportunities & Risks C o n s t r a i n t s M i s s & i o n E n v i r o n m e n t Uncertainty Shared Awareness Sensemaking Values Anticipated dynamic futures Alternatives perceived “Deep” understanding of situation Cause and effect Temporal relations Dynamic futures Was the right data collected? Was it put together appropriately? (correlation, context) Decision Support Models & Tools Was it put in a form that facilitates awareness? Did the individuals develop appropriate Situational Awareness (SA)? Was the SA shared? Was shared awareness of the situation correctly understood? Was Sense made of the situation? Emotions Physiological Factors Beliefs Perceptions Prior knowledge Mental Models Were the appropriate models and tools used? Were emotions, beliefs and cognitive factors taken into account? Were the decision and driving factors shared? Was command intent developed collaboratively? Was a quality plan developed? Was the plan executed effectively? Diagnosing Sensemaking

5. Situations Examined Historical Military: –Waterloo (1815) –Stonewall Jackson’s Valley Campaign (1862) –Lee at Chancellorsville (1863) –Gettysburg (1863) –Jutland (1916) –The Battle for France (1940) –Battle of Britain (1940) –Pearl Harbor (1941) –Sicily (1943) –Anzio (1944) –Normandy (1944) –Battle of the Bulge (1944) –Leyte Gulf (1944) –Monte Cassino (1944) –Vietnam Tet Offensive (1968) A total of 30 situations were examined:

6. Situations Examined (cont.) Contemporary Military –Vincennes shootdown of an Iranian airliner (1988) –F-15 shootdown of US helicopters over Northern No Fly Zone (1994) –Scott O’Grady shootdown over Bosnia (1995) Insurgencies –Mogadishu Ranger raid (1993) –Grozny (Chechnya) (1995) –Grozny (2000) Terrorism –King David Hotel bombing (1946) –Munich Olympics (1972) –Entebbe raid (1976) –Beirut Marine barracks bombing (1983) –Khobar Towers bombing (1996) –Attack on Japanese embassy in Peru (1996) –African US embassy bombings (1998) –USS Cole bombing (2000) –September 11 terrorist attacks (2001)

7. Analytic Groupings From these 30 situations, 149 specific decisions and decision processes were identified Classification of decisions and decision processes –93 historical military –10 contemporary military –8 insurgency –38 terrorism The 149 specific decisions were also grouped according to the success or failure of the decision –60 successes –89 failures

8. Research Issues Information System Inputs: –Was the right data collected? –Was it put together appropriately (correlation, context)? –Was it put in a form that facilitated awareness? Situational Awareness: –Did the individuals develop appropriate SA? –Was the SA appropriately shared? Cognitive Factors: –Were emotions, beliefs, and cognitive factors taken into account? –Was prior knowledge incorporated and used? –Were mental models used? Understanding: –Was shared awareness of the situation correctly understood? Sensemaking: –Was Sense made of the situation? Decision Effectiveness: –Was the chosen decision one that worked? Command Intent: –Was command intent developed collaboratively? Plan: –Was a quality plan developed? Execution: –Were the decisions and driving factors shared? –Was the plan executed effectively? Fifteen Sensemaking elements were grouped into 9 decision process groups:

9. Scaling Each element in each decision process was initially scored on a five-point scale: 5 - Yes, thoroughly 4 - Quite well, but not thoroughly 3 - Average, basics were covered 2 - Some, but not sufficiently 1 - No, not at all However the Cognitive Inputs were not reliably reported even in excellent sources. Therefore, the importance of these factors was rated, not their correctness in the cases. 51 Very Important Not Important

10. Cognitive Inputs Emotions, beliefs, cognitive factors Mental Models Prior knowledge Strong indication that cognitive inputs were important across all examples Prior knowledge was relatively less important than: – emotions, beliefs and cognitive factors – mental models x  4.25 1.00 3.87 1.34 3.19 1.42 To what extent were emotions, prior knowledge, and mental models important?

11. x  3.64* 1.35 3.37 1.45 3.37 1.46 3.31 1.41 Best Elements Average over all issues: What tended to go right? –Right data collected –Data put together appropriately –Decision and driving factors were shared –Information was put in a form that facilitated awareness Information systems generally tended to do their job 3.28 1.08 * Significantly different from the overall mean

12. Worst Elements Average over all issues: What tended to go wrong? –Collaboration –Effectiveness of decision –Effectiveness of execution x  2.57* 1.32 2.78* 1.65 2.90* 1.65 3.28 1.08 * Significantly below the overall mean

13. x  3.17 1.51 3.17 1.38 3.15 1.58 3.15 1.49 3.07 1.57 Typical Elements Average over all issues: –SA appropriately developed –SA appropriately shared –Quality plan developed –Shared awareness correctly understood –Sense was made of the situation 3.28 1.08

14. What Most Strongly Differentiated Successes From Failures? Did the individuals develop appropriate SA? Was Sense made of the situation? Was the data put together appropriately? Was the information put in a form that facilitated awareness? Were the decision and driving factors shared? Was shared awareness correctly understood? x suc - x fail 1.86 1.82 1.70 1.69 1.59 1.58

15. Conclusions Cognitive inputs are important elements and must be addressed for successful military Sensemaking Best overall performance tended to be things within the information systems, or supported by the information systems Weakest elements tended to be collaborative processes and ability to synchronize Differentiation between success and failure tended to focus on situation awareness, understanding, Sensemaking, and sharing decisions and driving factors Analysis continues...