1. Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr. Ute Fischer Sponsored by the FAA, Tom McCloy as Technical Monitor DEVELOPING A TRAINING PROGRAM FOR THE TRAFFIC ALERT AND COLLISION AVOIDANCE SYSTEM IN CONTEXT

2. Overview +Introduction to TCAS and Background Information +Design of Training Program +Evaluation of the Training Program +Results +Conclusions 2 Develop a training program intended to improve pilots’ understanding of TCAS use for collision avoidance in a range of traffic situations

3. Traffic alert and Collision Avoidance System (TCAS) 3 +Pilot always shown the Traffic Situation Display (TSD) +TCAS delivers a two stage advisory and vertical avoidance maneuver  Traffic Advisory (TA) - ‘Traffic Traffic’  Resolution Advisory (RA) - ‘Climb Climb’ +Pilot is to follow an RA, even if it conflicts with ATC instructions, unless the pilot believes that safe flight would be jeopardized Federal Aviation Administration (2000). Introduction to TCAS II Version 7. Washington, D.C.

4. Collision Avoidance in a Broader Context Pilots don’t only interact with TCAS, they also receive information from ATC and the environment 4 Pilot’s Response Advised Maneuvers for Collision Avoidance Notification and Awareness Awareness of other aircraft via TSD TCAS Traffic Advisory TCAS Resolution Advisory ATC Traffic Callout Visual acquisition of other aircraft ATC Maneuver Compliance to TCAS Compliance to ATC Personal Assessment and Maneuver Decision Party-line Information

5. Current TCAS Training Requirements: Ground Training +Classroom or computer based +>60 training requirements +Measures performance through quizzes and/or activities 5 To comply to the RA, you should Pull the stick back Push the stick forward Federal Aviation Administration (2001). Advisory Circular No. 120-55B: Air Carrier Operational Approval and Use of TCAS II. Washington, D.C.

6. Current TCAS Training Requirements: Flight Training 6

7. Observed TCAS Use and Operation o The TCAS Operational Performance Assessment (TOPA) monitored the terminal area of 8 major airports and examined pilot compliance to climb and descend advisories 7 41% to Climb RA’s 59% to Descend RA’s TOPA observed compliance rates of Misunderstanding/ Confusion Aggressiveness Non-Compliance Olson, W. and J. Olszta (2010). TCAS Operational Performance Assessment in the U.S. National Airspace. IEEE/AIAA Digital Avionics Systems Conference. Olszta, J., & Olson, W. (2011). Characterization and Analysis of Traffic Alert and Collision Avoidance Resolution Advisories Resulting for 500' and 1,000' Vertical Separation. Paper presented at the Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011), Berlin, Germany.

8. Observed TCAS Use and Operation 8 “[My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a person should fly 'away' from the red VSI indication, if instructed via RA.” ACN:785761, 2008 “[My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a person should fly 'away' from the red VSI indication, if instructed via RA.” ACN:785761, 2008 Pilot reports flying into the red region on the VSI Non-Compliance Aggressiveness Misunderstanding/ Confusion NASA. (2009). Aviation Safety Reporting System. Retrieved August 1, 2010: http://asrs.arc.nasa.gov/ “Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the initial guidance was so clearly wrong” ACN: 854982, 2009 “Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the initial guidance was so clearly wrong” ACN: 854982, 2009 Pilot disagrees with advised descend RA

9. Observed TCAS Use and Operation 9 Non-Compliance Misunderstanding/ Confusion Aggressiveness Far Eastern Air B757 Response to Descend RA (TCAS advised a descent rate of 1500 FPM) Image copied from Lacagnina (2008). Easy Does It. Aero Safety World : http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1

10. Research Statement and Objectives Develop a training program intended to improve pilots’ understanding of TCAS use for collision avoidance in a range of traffic situations 10 (1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment (2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories (1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment (2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories

11. 11 Demonstration Based Training Event Based Training Approach to Training Design Complement of Two Methods

12. Demonstration Based Training (DBT) +Computer-Based Training  ~25 minutes  Provides conceptual understanding of TCAS  Outlines rules for compliance +6 segments  Introduction to TCAS  Traffic Situation Display  TCAS Advising Logic  Traffic Advisories  Resolution Advisories  Example Timeline of RA Evolution +Mid-Training Quizzes 12

13. DBT: Demonstrations of RA’s 13

14. Event Based Training (EBT) +Presents traffic events that create the requirement to act +Builds context and complexity into each scenario as the flight progresses +Uses a more realistic training environment Fowlkes, J., Dwyer, D., Oser, R., & Salas, E. (1998). Event-Based Approach to Training (EBAT). The International Jounal of Aviation Psychology, 8(3), 209-221. 14

15. Air Traffic Transcripts Charts & Checklists Audio Communications (Aviation Intercom) TCAS Alerts Simulation Architecture Eyetracker PartyLine First Officer Captain Experimenter/Instructor Coded Log of Flights B747-400 Simulator (RFS) TSD PFD ND PFD ND Touch screen SideStick ATC->TSD VGA ATC Air Traffic Simulator (TGF) TCAS Logic 15 Simulator Study in Integrated Flightdeck – ATC Environment Pritchett, A., Fleming, E., Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012). Pilot Interaction with TCAS and Air Traffic Control. Paper presented at the 2nd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS), London.

16. Structure of Events: Descend RA Example 16 Training Objective Context of EventPerformance Measures Feedback Session Accurate interpretation of and response to TCAS Descend RA. Instrument Meteorological Conditions (daytime, clouds, no winds) ATC provides no traffic information Conflict caused by IFR traffic enroute (most likely on departure) RA maneuvering should not violate ATC instructions No conflicting ATC or party-line information Pilot disengages autopilot and flight directors If the pilot did not meet any particular performance measure, review the correct response in regards to that measure Pilot responds to advisory with appropriate vertical speed Pilot ensures vertical speed is not excessive o Pilot notifies ATC of response to TCAS advisory as the maneuver is performed Pilot reengages autopilot and flight directors o Pilot notifies ATC of TCAS advisory and response after clear of conflict Pilot returns to original clearance (if needed)

17. Evaluating the training program’s impact

18. +Baseline Study +Training Study +6 scenarios with 2 traffic events per scenario +Traffic events defined by  RA type  ATC information  Traffic density Overview of Study 18 Comparison to identify the impact of training

19. Overview of Study Pre-Training Data Collection (50 minutes) Pre-Experiment Questionnaire Pre-Experiment Quiz TCAS Training Program (50 minutes) Introduction to TCAS Demonstration Based Training Short Section Quizzes Event Based Training Evaluating TCAS Training Program (120 minutes) Flight Scenarios Post Scenario Questionnaires Debrief (15 minutes) Post- Experiment Questionnaire 19

20. Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior 20 Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?

21. 21 Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?  Decrease time to achieve compliance  Decrease time to disconnect autopilot  Reduce aggressive response features  Increase percentage of RA duration in compliance  Increase appropriate response post-Clear of Conflict  Increase understanding of TCAS  Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

22. Assessing the Pilot’s Response: Skill-Based Behaviors 22 Vertical Speed time 5 seconds RA Climb 2 ½ sec TCAS weakens required vertical rate TCAS assumed ¼ g pull-up Clear of Conflict TCAS assumed constant vertical RA rate Autopilot Disconnect Time TCAS RA Maneuver Pilot’s Response Time Pilot First Achieves Compliance Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior X

23. Mean Time to Comply: During Training 23 Skill-Based Behavior p MM = Significance for the mixed model p  2 = Significance of the variance p  = Significance of the means Rule-Based Behavior Knowledge- Based Behavior

24. +During training, no significant differences +Post training  Autopilot disconnect time decreased  Pilot response was more consistent for one event Autopilot Disconnect: Post Training 24 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

25. Impact of Training Program 25 Decrease time to achieve compliance Decrease time to disconnect autopilot  Reduce aggressive response features  Increase percentage of RA duration in compliance  Increase appropriate response post-Clear of Conflict  Increase understanding of TCAS  Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior Skill-Based Behavior Rule-Based Behavior

26. Assessing the Pilot’s Response: Aggressiveness 26 Vertical Speed time 5 seconds RA Climb 2 ½ sec TCAS weakens required vertical rate TCAS assumed ¼ g pull-up Clear of Conflict TCAS assumed constant vertical RA rate Maximum vertical rate Maximum vertical rate difference Vertical rate difference TCAS RA Maneuver Pilot’s Response Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior *Altitude Deviation

27. Assessing the Pilot’s Response: Compliance 27 Vertical Speed Not in Compliance In Compliance TCAS RA Maneuver Pilot’s Response time 5 seconds RA Climb 2 ½ sec TCAS weakens required vertical rate TCAS assumed ¼ g pull-up Clear of Conflict TCAS assumed constant vertical RA rate Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

28. Assessing the Pilot’s Response: Return to Clearance 28 TCAS RA Maneuver Pilot’s Response TCAS assumed constant vertical RA rate TCAS weakens required vertical rate Clear of Conflict Altitude time 5 seconds RA Climb TCAS assumed ¼ g pull-up Cleared altitude at time of RA Pilot holds new altitude Contacts ATC and request new clearance or ask for further instructions Pilot begins descent back to originally cleared altitude May inform ATC of response to RA Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

29. Before Training +89% of the pilots did not know that an RA should cause less than 500 feet of altitude deviation +56% of the pilots responded that they would hold current altitude achieved after responding to an RA, as opposed to returning to their clearance +When asked about airline procedures for following TCAS, all 18 pilots noted the need for complying with an RA  BUT 28% of the pilots commented compliance wasn’t necessary is there was a TCAS “malfunction” or if the RA would cause an “unsafe situation” 29 “[Pilots] must always comply with a TCAS RA unless [aircraft] performance is hindered (i.e. operating single engine) or [there is] an obvious TCAS malfunction (ie you can see traffic and it is not a threat)” Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

30. +During training, aggressive response features decreased +Post training, same decreasing trend observed Aggressiveness 30 Altitude Deviation Average Vertical Rate Difference Maximum Vertical Rate Difference All decreased Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

31. +During training, percentage compliance decreased in training event with “Climb RA” (93.1% compared to 99.6%) +Post training  No significant differences in means observed  But, trained pilots had a more consistent response Percentage Compliance 31 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

32. Return to Clearance 32 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior Pilot returns to original clearance No Yes

33. Impact of Training Program 33 Decrease time to achieve compliance Decrease time to disconnect autopilot Reduce aggressive response features  Increase percentage of RA duration in compliance Increase appropriate response post-Clear of Conflict  Increase understanding of TCAS  Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior Rule-Based Behavior Knowledge- Based Behavior

34. Before Training +94% pilots agreed with statement “I understand TCAS maneuvers when they are issued”, but…  Only 50% of the pilots correctly identified the assumptions made by TCAS advisory logic  When asked to interpret TSD symbols, only 50% of the pilots got all parts of the associated questions correct 34 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

35. “My understanding of TCAS has increased” +50% pilots reported an increase in understanding TCAS logic +27% pilots claimed to have learned about different types of TCAS RA’s (notably, “Crossing RA’s”) Post Training Understanding of TCAS 35 “Types of RA's were not previously taught. We were taught simply to comply” Strongly Disagreed 0 0 Neutral 6% Agreed 61% Strongly Agreed 33% Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

36. Post Training Trust in TCAS “I am more likely to trust TCAS after completing today’s training” 36 “My trust in TCAS was already at a maximum so I wouldn’t be ‘more’ likely to trust it” Strongly Disagreed 11% Disagreed 0 Neutral 39% Agreed 39% Strongly Agreed 11% Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

37. Post-Training 37 Decrease time to achieve compliance Decrease time to disconnect autopilot Reduce aggressive response features  Increase percentage of RA duration in compliance Increase appropriate response post-Clear of Conflict Increase understanding of TCAS Increase trust in TCAS

38. Conclusions and Future Work Conclusions +Pilots may not need more training, but instead need better training  Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive  EBT structure allows for the design of purposeful training events +Current FAA mandated TCAS training objectives may not fully reflect all training areas  Language needed to address the reduction of excessive responses to RA’s Future Work +What facilities and technologies would be required to implement this type of training program? +What implications arise when considering training design versus system design?  Would incorporating human factors considerations in initial design stages decrease the amount of required training? 38

39. 39

40. Conclusions and Future Work Conclusions +Pilots may not need more training, but instead need better training  Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive  EBT structure allows for the design of purposeful training events +Current FAA mandated TCAS training objectives may not fully reflect all training areas  Language needed to address the reduction of excessive responses to RA’s Future Work +What facilities and technologies would be required to implement this type of training program? +What implications arise when considering training design versus system design?  Would incorporating human factors considerations in initial design stages decrease the amount of required training? 40

41. Acknowledgements +Work sponsored by the FAA, Tom McCloy as Technical Monitor +34 pilot participants +Dr. Amy Pritchett +Dr. Karen Feigh +Dr. Ute Fischer +Dr. Wesley Olson, MIT Lincoln Labs +Wayne Gallo, FAA +Roger Sultan, FAA +Kylie Garey +TCAS Team: William Cleveland, Vlad Popescu, Justin Mullins, Anil Bozan, Henry Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, Dhruv Thaakar, Jonathan Zoetrum, Jelle Wissink +CEC Lab Members 41

42. 42