1. Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and Astronautics http://halab.mit.edu

2. Review of Original Proposal Project Goal  Develop technological support for futuristic unmanned aerial vehicle (UAV) teams engaged in TST operations (but really any time critical resource allocation task) Technology to Support the Team Supervisor: Activity Awareness Displays Interruption Recovery Tools Technology to Support UAV Operators: Activity Awareness Displays Objective Performance Measures

3. Teamwork in UAV Operations Intelligence Consumers (e.g., Ground Troops) UAV Operators Operations CenterGround Crew Vehicle(s)

4. Generalizable to Other Domains 3 “Confederates” as UAV Operators

5. Team Interface Issues in Time-Critical Operations Increasing reliance on collaboration tools -e.g., email, instant messaging (chat) -Increased communication overhead Need interfaces that intelligently share activity information to improve planning & coordination in networked teams -Activity awareness Research for technology aids to assist supervisors of teams virtually non- existent Measurement of technology interventions very difficult -Objective team performance metrics are elusive

6. Boeing Team Test Facility 3 “Confederates” as UAV Operators

7. Simulated Task Environment Simulated Task Environment Situation Map DisplayMission Status Display Remote Assistance Display Surveillance support for a ground convoy through a hostile region UAV team consists of: -1 Mission Commander -3 UAV operators, each controlling multiple UAVs UAVs have camera sensors only -Team must coordinate with external strike team to destroy identified threats

8. Team Supervision Decision Aiding UAV Operations Team Interruption recovery tools for the team supervisor

9. Interruption Recovery Assistance (IRA) tool Interactive event timeline, containing event “bookmarks” Increased recovery time but positive impact on decision accuracy, especially in complex task situations. IRA tool tended to provide greater benefits to participants without military experience Some experimental/interface issues so additional study needed Summer 2008 w/ Waterloo

10. Current Work: Predictive Tool for Team Supervision UAV Team Supervisor Technologies for assessing: -How well is the team doing? -When & who should received attention? Team

11. The need for more objective team performance metrics -Our focus is supervisory control in team settings with embedded layers of automation Predicting team states versus actual performance -Artificial intelligence, behavioral pattern detection and performance correlations Alert team supervising agent of sub-optimal team behavior/cognitive strategies -Relying on human knowledge-based reasoning to determine if an anomalous team state exists Two stage approach -Predict an individual operator’s state transition as a proof of concept -Adapt model to UAV team setting Current Work: Team Performance Prediction

12. Evaluation and predictions of team behaviors: -Automatic, continuous, and in real-time Only relies on easily observable data -User interaction with machine -User communication -Eye tracking? Use Bayesian pattern prediction methods applied to individual/team behaviors -Hidden Markov Model Not predicting performance per se -Anomalous conditions -Value of human judgment Team Prediction Model Characteristics Team Prediction Model Characteristics

13. Probabilistic state transitions Observables vs. Hidden States - Hidden states are not directly visible, but variables influenced by them are. - Cognitive states vs. behaviors HMM uses the observables to infer: -Most likely hidden state sequence -Based on future sequences, the most likely observables -Machine learns the clustering of behaviors (need lots of data!) Hidden Markov Models Modify Flight Parameters Comms Camera Control Waypoint Update H&S Monitoring Target Monitoring Mission Replan

14. Predicting Individual Behavior

15. Human State Estimation - TRACS

16. Building the Model Grammar Pattern Recognizer/ Predictor Low Level Input Future Team Behavior ILDs

17. 4 State HMM for Mission Planning Task Unsupervised vs. supervised model training for state recognition Leveraging Automation Manual Planning Exploration

18. Predicting an Operator’s Strategies

19. Live Demo!

20. Validating our HMM Approach Currently only behaviors that entail mouse clicks are observed -A limit, particularly for monitoring interfaces Eye tracking arguably gives us additional data about the information operators access -Noisy -Cost-benefit analysis? Study will be conducted this summer comparing HMM results with and without eye tracking input

21. Extending our Approach to UV Teams Visualization task Evaluation/ Choice of UV + engage Choice of UV Sector A Sector B Sector C a) Replan Path b) Replan Target Extending to task to teams: UV hand-off between Sectors

22. The Plan Develop HMM using experimental data from individual experiments -Investigate supervised vs. unsupervised learning Develop a team version of the interface Update HMM with team experiments to be conducted this summer

23. Additional Future Work: Designing for Team Activity Awareness UAV Operations Team

24. Preliminary UAV Operator Display Designs Map Display Communications Display Tasking Display: TargetID & rerouting, reassigning UAVs

25. Deliverables 6 Conference papers - ASNE, HSIS, ICCRTS, HFES 4+ Technical reports 3 Undergraduate theses - 2 pending 1 Masters thesis 1 PhD dissertation (pending) 3 Journal articles in various stages 1 workshop (CSCW 2006) IP disclosure pending HMM validation The Holy Grail of team research???

26. Near Term -Eye tracking study -HMM-Multi UV experiment -Extension for supervisor IRA tool Mid Term -Activity awareness displays for operators -Decision support display for HMM -Multi-UV connection to predictive model system architecture research (ONR)  Development of CE metric Project slated to end December 2008 -Requested no-cost extension to JUN 09 -Additional funding Future Plans

27. Backup

28. Simulated Task Environment Display Detail Map Display Mission Status Display

29. Simulated Task Software Architecture Simulated Task Software Architecture Large-Screen Wall Displays Situation Map DisplayMission Status Display TabletPC Display Mission Commander Display Simulation & Collaboration Server (Grouplab SharedDictionary)

30. Situation Map Display Mission Status Display Designing to Promote Activity Awareness UAV status & tasking Current & expected convoy safety Current & expected operator task performance, relative to convoy safety 4 (nominal) (down) 8 1 (reviewing ATR imagery)

31. Designing to Promote Activity Awareness Potential threat envelope Target strike indicators Known threat envelope Situation Map Display Mission Status Display UAV status & tasking Current & expected convoy safety Current & expected operator task performance, relative to convoy safety

32. Situation Map Display Mission Status Display Designing to Promote Activity Awareness UAV status & tasking Current & expected convoy safety Current & expected operator task performance, relative to convoy safety

33. Developing Methodology for Deriving Collaborative System Requirements Project Goal: To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking