 2009, Aptima, Inc. 1 MA ▪ DC ▪ OH ▪ FL © 2009, Aptima, Inc. Human-Centered Engineering Perspectives on Simulation-Based Training Daniel Serfaty Emily Wiese Presented to SimTrans Copenhagen, Denmark 22 June 2009

 2009, Aptima, Inc. 2 Agenda  Introduction to Aptima –Examples of Capabilities in Human-Centered Engineering  Five Emerging Technologies in Simulation-Based Training –Scenario Engineering –Simulation Fidelity –Performance Measurement (with A-Measure toolkit demo) –Cognitive Skills Training –Team Communications Assessment  Discussion

 2009, Aptima, Inc. 3 What is Human–Centered Engineering? Technology Capabilities Social & Organizational Structures Mission, Tasks & Work Processes Human Agents Congruence

 2009, Aptima, Inc. 44 Aptima, Inc.  Interdisciplinary Small Business –Founded in 1995 –Consistent annual growth (40% CAGR) –100+ staff (80% graduate degrees)  Human Centered Engineering –Analyze and design complex socio-technical systems –Combine social science theory with quantitative, computational methods  Serving government and commercial clients –350+ contracts with the Defense Industry  Offices –Boston/Woburn, MA, (HQ) –Washington, DC –Dayton, OH –Ft Walton Beach, FL

 2009, Aptima, Inc. 55 Domain Expertise Skill Set  Command & Control  Military Training  Leadership  Complex Information Display  National Security Solutions  Medical & Healthcare  Aviation  Emergency Preparedness  Stability & Support Operations  Education  Safety Educational Backgrounds

 2009, Aptima, Inc. 6 Optimizing Performance in Mission- Critical Environments

 2009, Aptima, Inc. 7 Examples of Capabilities  Performance Measurement  Socio-Cultural Applications  Organizational Engineering  Training

 2009, Aptima, Inc. 8

9

10 Official U.S. Navy photo. Neither the U.S. Navy nor any other component nor any other component of the Department of Defense has approved, endorsed, or authorized this product [or promotion, or service, or activity].

 2009, Aptima, Inc. 11

 2009, Aptima, Inc. 12

 2009, Aptima, Inc. 13

 2009, Aptima, Inc. 14

 2009, Aptima, Inc. 15

 2009, Aptima, Inc. 16

 2009, Aptima, Inc. 17

 2009, Aptima, Inc. 18

 2009, Aptima, Inc. 19

 2009, Aptima, Inc. 20

 2009, Aptima, Inc. 21 Agenda  Introduction to Aptima –Examples of Capabilities in Human-Centered Engineering  Five Emerging Technologies in Simulation-Based Training –Scenario Engineering –Simulation Fidelity –Performance Measurement (with A-Measure toolkit demo) –Cognitive Skills Training –Team Communications Assessment  Discussion

The New Science of Scenario Engineering  BEST –Engineering the Stimulus for Optimal Learning  PRESTO –Optimizing Learning Trajectories Using Constraint-Based Logic  CROSSTAFF –Engineering Training Scenarios from Operational Data  VSG –DDD Visual Scenario Generator Tool Q: How to Optimize Learning on a Given Simulator?

Vision: Learning from Synthetic Experiences Training Opportunities Training Experiences Next Scenario Readiness Competencies, Knowledge, Skills… Training Objectives What to Measure Performance Measurement Data

BEST: Optimizing Scenario Selection  Assess team performance against near-optimal solution  Based on that assessment, select training events to optimize the team’s learning curve Training Session Team Score What sequence of experiences moves a team to a steeper learning curve? System’s Belief about State of Competencies True State of Competencies s(n-1) s(n) s(n+1) X(n-1) X(n) X(n+1) scenario at n scenario at n reward at n reward at n observation at n observation at n scenario at n+1 scenario at n+1 reward at n+1 reward at n+1 Parameters set by SMEs Tool: Partially Observable Markov Decision Process (POMDP) Training Model

Experiment Result: 52% Improvement over Best Practices Team Training

Conventional vs. PRESTO-Based Scenarios Conventional Scenario MSEL Events Planned Actually Occurred Didn’t Occur PRESTO-based Scenario Actual PRESTO-Based Scenario Space

(a) Original Flight Path(b) Key Events Identified (d) Scenario Envelope Generated(c) Events Generalized Engineering Training Scenarios from Operational Data (CROSSTAFF)

Scenario Engineering Using the DDD* Visual Scenario Generator (VSG) DDD*: Distributed Dynamic Decision-making simulator

Pedagogically-focused Adaptive Scenarios Scenario Initial Conditions Participant Advice Exercise Events Training Objectives TO Conditions Ongoing Training Exercise PRESTO/ CROSSTAFF SAF/Instructor Advice Performance Measurement System (A-Measure) Trainee Performance Measures BEST Optimal Vignette Sequencing

Understanding Simulator Fidelity Requirements 1. There is little guidance and no standard tool for determining the appropriate level of fidelity of training simulators to –Achieve specified training objectives, –Maintain trainee acceptance, and –Fit within budgetary constraints. Simulator Fidelity Training Effectiveness 2. There are no standard measures designed to be sensitive enough to detect objective performance differences invoked by varying levels of fidelity Perceived Actual Cost  2009, Aptima, Inc.

31 RELATE: A Research-Driven Approach  A systematic approach to establish quantitative, predictive relationships between simulator fidelity and training effectiveness  RELATE fuses: –Fidelity requirements defined by end-users; –Existing theory and research about fidelity; and –Objective performance data from fidelity experiments to develop a predictive, computational model.  2009, Aptima, Inc.

32 Model-Based Tool  A computational model-based tool to assist with decisions regarding the acquisition and use of training simulators  Tool can help users:  Conduct return on investment analyses to determine which simulator to develop or acquire  Prioritize technology enhancements to improve the effectiveness of existing simulators  Develop an strategy for employing both high- and low-fidelity simulators to meet training objectives

 2007, Aptima, Inc. 33 PERFORM: Air-Air Combat Fidelity Requirements  Conducted research to examine the visual and cockpit fidelity requirements for training air-to-air combat skills to experienced pilots in F-16 simulators  Developed a decision-support tool to help the Air Force prioritize technology enhancements to improve the effectiveness of deployable simulators Deployable Tactics Trainer (DTT) Display for Advanced Research and Technology (DART)

34 PREDICT: Air-Ground Combat Fidelity Requirements  Conducting research to examine the visual fidelity requirements for training air-to-ground combat skills to inexperienced pilots in F-18 simulators  Developing a decision-support tool to help the Navy develop a strategy for employing both high- and low-fidelity simulators to meet training objectives  2009, Aptima, Inc.

35 FLEET: Developing and Validating Fidelity-Sensitive Measures  Developing measures of pilot performance that are sensitive enough to detect objective performance differences invoked by varying levels of fidelity  2009, Aptima, Inc. Fidelity-Sensitive Performance Measures Does the pilot complete Combat Fence IN Checks? Does the pilot maintain briefed formation? Does the pilot appropriately mitigate surface-to-air threats? Does the pilot deconflict with other assets?  Conducting research to examine the motion fidelity requirements for training air-to-ground combat skills to inexperienced pilots in T-45 simulators

Summary  Aptima seeks to identify the simulator fidelity requirements for effective training by:  Developing a systematic approach to establish quantitative, predictive relationships between simulator fidelity and training effectiveness  Building a computational model-based tool to predict the impact of simulator fidelity on training effectiveness  Creating performance measures and measurement tools that can be used to collect better data in simulator fidelity experiments  Employing the proper level of fidelity will ensure better training results and reduce costs by eliminating investments in unnecessary training and technology  2009, Aptima, Inc.

New Approaches to Measuring Performance  A well-designed measurement system makes simulation-based practice effective training –The right feedback to the right person at the right time leads to better learning  Measurement enables assessment of training effectiveness –Are people getting the skills they need?  Guides selection of training environment –Live, virtual, constructive –Facilitates appropriate use of measures  Measurement technology can turn simulators into training machines

You Can’t Train What You Can’t Measure  Why is this hard? –Volume of data –“ We recorded everything...” –Lack of meaningful aggregation methods –“325,435 messages were received…the average length was 2.35 minutes” –Interdependence of behaviors at different locations –No one person has the total picture –“Correct” behavior depends on dynamic context –It is hard to construct, even after the fact, where the team went wrong Hours spent training ≠ Proficiency Real-world experience ≠ Proficiency

Performance Measurement Process

Competency Based Performance Measurement  Competency-based performance measures  Leverages performance measurement theory in combination with subject matter expert input  Assesses team and individual performance The COMPASS SM Methodology is a product of Aptima, Inc. COMPASS SM tells us the what direction to go with measure development

A▪Measure Product Family Turning simulators into training machines

Cognitive Skills Training Not all skills should be trained the same way –Cognitive skills vs. procedural skills Pedagogical Theory: Direct instruction vs. constructivism  Most types of training/education employ a direct instruction approach –Can be effective for training procedural skills  Current research suggests that a problem-based approach may be more appropriate for training cognitive skills –What differs is when you tell someone how to do something relative to when they practice doing it  E.g., telling students the formula for density and having them practice (the traditional approach) enforces a plug-n-play understanding. Very little transfer.  Instead, give students a situation and ask them how they would describe density. –Now they get a sense of the principles involved before you give the formula.

The Bransford Model

Aptima’s Balanced Unified Incremental Learning Development (BUILD) Training Approach

Example: How to “train the ear”  Air Battle Managers (ABMs) monitor multiple communications (radio) channels at the same time  This currently an acquired skill developed through experience and on the job training.  How can we train a novice?  At its core, this is a skill that relies heavily on cognitive skills like memory and attention.  Monitoring multiple communications channels requires… –Dedicating limited cognitive resources (memory) –To attending to stimuli that must pass a certain threshold (attention) –Under stressful conditions (stress)

How can these cognitive concepts help ABMs?  “Monitoring multiple comms channels requires… –Dedicating limited cognitive resources (memory)”  Psychological research demonstrates that one can free limited working memory resources by placing some information in long term memory.  The key to long term memory storage is automaticity. –When I see X, I do Y  The key to training automaticity is repetition –Multiple trials

How can these cognitive concepts help ABMs?  “Monitoring multiple comms channels requires… –Dedicating limited cognitive resources (memory)” –To attending to stimuli that must pass a certain threshold (attention)”  The threshold is often physical (a certain volume, a certain brightness), but could also be semantic (meaningful). –Cocktail party effect  Trigger words can break the semantic threshold for attention –When I hear an important word, I attend to it  This is the mechanism for retrieval.

From Theory to Application: A Layered Comms Training Approach  Phase I: Trainees are introduced to ABM trigger words and their definitions. –Begin to store in long term memory  Phase II: Trainees recognize trigger words within a stream of communications. –Develop automaticity and retrieval  Phase III: Trainees recognize a trigger word in a realistic scenario and respond accordingly. –Automatically retrieve under stressful conditions

Cognitive Skills Training

CIFTS: Communications Analysis in Operational Environments  2009, Aptima, Inc. 50  Domain: Air and Space Operations Center (AOC)  Numerous centers around the world  Around 100 operators communicating  In the air  On the ground  Around the world  Extremely complex operations must be coordinated  Extensive use of chat to coordinate, assign tasks, exchange information “Communications is at the Heart of Team Performance”

CIFTS Project Domain: AOC Chat Data  2009, Aptima, Inc. 51 [23:09] TDO: still orthanc where unsuccessful [23:09] firing unit how can rtb green up [23:09] tdn shot down fac [23:09] SOLE: relay to inform pol leadership lost [23:10] current position standby dtl updated [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] TGTS: please might mean [23:12] sir bent no [23:13] high fast flyer affirm [23:13] c [23:09] still orthanc where unsuccessful [23:09] firing unit how can rtb green up [23:09] tdn shot down fac [23:09] GTC: relay inform pol leadership lost [23:10] current position standby [23:09] TDO: still orthanc where unsuccessful [23:09] firing unit how can rtb green up [23:09] tdn shot down fac [23:09] SOLE: relay to inform pol leadership lost [23:10] current position standby dtl updated [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] TGTS: please might mean [23:12] sir bent no [23:13] high fast flyer affirm [23:13] c [23:14] Chief: attack check jdocs sado right wrkg unfriendly lost track [23:14] SOLE: anyone did you copy can you confirm [23:15] ATK: what up [23:16] still issues [23:16] TDO: wrking wrong sam ring return to base facilities who is disgard [23:16] SODO: jadocs each time wrong voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:17] GTC: facility oga rotary wrkg [23:18] resend [23:21] not good wmd vehicles where [23:22] wmd vehicles ukn roll call requested do u agree [23:23] GTC: all c2 players do u launch pads rqist control measure was back up [23:23] no idea [23:25] SADO: strike asset changing cco [23:25] negs cvy need no pred [23:25] what were you doesnt vehicle assembly msn results [23:25] ATK: got it mass of vehicles pimp Need automated methods to understand what is happening in general, and the ability to drill down to specific instances

Advanced Language Analysis: LAVA TOOLkit  LAVA is Aptima’s LAtent Variable Analysis toolkit  LAVA provides tools for natural language processing –Processing free text to represent words as numbers –Statistical tools to extract concepts from free text  LAVA is language-independent –Inter-agency, inter-cultural, international differences  The kinds of questions you can ask of LAVA –What are the main concepts within this set of news articles? –How similar are these two medical records? –What is this talking about? –How does the subject change over time?  LAVA in JAVA –Intel Math Kernel Library –Microsoft SQL Server or MySQL –Web services –JAVA API 52  2009, Aptima, Inc.

PACE: Processing and Analysis of Communications and Events  Addressee  Density –High Density  Message Type –Question –Acknowledgment –Command –Ambiguity –Pause  Valence –Positive –Negative  Mission ID –JA0002 == Underground Bunker  Chains –GTC – TDO  Combinations….  2009, Aptima, Inc. 53 [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn [23:09] TDO: still orthanc where unsuccessful [23:09] SOLE: can we green up? [23:09] c, fac shot down [23:09] SOLE: relay to inform pol leadership lost [23:10] current position JA0002 standby dtl [23:11] TGTS: thinks minus one chem fac why [23:12] TGTS: hvaa retrograde transport type [23:12] what does hvaa mean? [23:12] sir bent no [23:13] please confirm high fast flyer [23:13] c [23:13] Chief: attack JA0002 check jdocs [23:14] TDO: did you copy? [23:15] sb [23:16] still issues [23:16] TDO: wrking return to base JAoo2 [23:16] SODO: jadocs each time voice comms [23:17] kill call underground bunker correct atk [23:17] jstars armored car link track [23:18] GTC: facility oga rotary wrkg [23:18] cpy [23:21] GTC: where are wmd vehicles? [23:22] wmd vehicles ukn

Non-Combatant Evacuation Data Example: Valence and Type  2009, Aptima, Inc. 54

CIFTS Timeline Interface  2009, Aptima, Inc. 55 Tick marks indicate messages in timeline. Colors indicate values for the current analysis. Any two analyses can be “crossed” Messages can be filtered by checking values on each analysis type Tabs display different analyses Content of messages Tabs hold different kinds of summary data

CIFTS Network Interface  2009, Aptima, Inc. 56 Arrow A->B indicates message from A was followed by a message from B within the same chat room; on average, a “response” Arrow thickness indicates % of messages from A that were followed by messages from B; “conversations” Size of the circle indicates percent of all messages that this member sent Length of the line indicates the time interval between messages; “density” of comms Layout tries to minimize line- crossings; localizes “functionality” More “central” members have contacts with more other members

Conclusions & Questions  Key Questions: –In a world dominated by simulation-based training, how do we optimize learning yield (ROI)? –In scenario-based training, what is the “curriculum”? –How can we get effective and efficient in performance measures selection and feedback? –How can rigorous scientific methods help contribute to the above? –Scientific methods  Practical software tools?

Daniel Serfaty: Emily Wiese: