1. MIT ICAT MIT ICAT Supporting Pilots’ Time-Dependent Information Needs During Operations in Adverse Weather L aurence Vigeant-LangloisR. John Hansman, Jr. langlois@mit.edulanglois@mit.edu rjhans@mit.edurjhans@mit.edu

2. MIT ICAT MIT ICAT Motivation Adverse Weather Significantly Impacts Flight Operations Safety -- 22. 5% All US Accidents Efficiency -- 17% / $1.7B per year Avoidable Weather Delays (Source: FAA) Multiple efforts to develop new weather information tools Cockpit weather datalink In order to develop safe and effective decision-support tools, it is important to: Understand users’ information needs and cognitive tasks Understand the implications and limitations of supporting their tasks Turbulence Convective Weather Icing (Courtesy of NASA) Ceiling & Visibility

3. MIT ICAT MIT ICAT Methodology Cognitive Analysis of Pilots Decisions Temporal Representation Framework Deterministic Regime Stochastic Regime

4. MIT ICAT MIT ICAT Human-Centered Approach Closed Loop Feedback Process Aircraft Sensors Numerical Wx Models Trajectory Control Direct Weather Observation Situation Dynamics Pilot Interaction Weather Aircraft State Displays Direct Aircraft State Observation Sensors Model Human Operator Displays Aviation Weather Information Remote Wx Sensors Sensor Readings Reports Fore- casts Voice Datalink In-Situ Wx Sensors Dissemination

5. MIT ICAT MIT ICAT 3 Projection DECISION Monitor Evaluate Adjust Select Project Understand Perceive Weather State Aircraft State Interaction SITUATIONAL AWARENESS Weather Phenomenology Aircraft Enveloppe Future Exposure Weather Forecast Aircraft Trajectory PLAN Formulate Nominal Plan Formulate Alternative Plans PERFORMANCE OF ACTIONS Implement Aircraft Situation Dynamics Interaction Weather INFORMATION Sensors Aircraft Trajectory Control Displays WORKING MENTAL MODEL Pilot

6. MIT ICAT MIT ICAT Route Selection Go/No-Go Tactical Avoidance Escape Situation Monitoring Diversion Non-Weather-Related Replanning Aircraft Systems Management Request for Weather Information Use of Emergency Authority Reactive Tactical Strategic Route Planning Planning Tasks of Pilots Weather-Related Decisions Time Short-term planning where safety goals dominate the decisions Planning for the remainder of the flight considering the main goals: safety, legality, efficiency, satisfactory level of comfort and service Planning based on the main goals without considering the entire remainder of the flight

7. MIT ICAT MIT ICAT Methodology Cognitive Analysis of Pilots Decisions Temporal Representation Framework Deterministic Regime Stochastic Regime

8. MIT ICAT MIT ICAT WEATHER REPRESENTATION HistoricalConstant Stochastic ? TIME Temporal Regimes of Cognitive Processes Time of Information Production ReactiveTactical Strategic FLIGHT PLANNING Time of Information Use Deterministic

9. MIT ICAT MIT ICAT Methodology Cognitive Analysis of Pilots Decisions Temporal Representation Framework Deterministic Regime Stochastic Regime

10. MIT ICAT MIT ICAT Pilots’ Perception of Forecast Quality Protected Aircraft Hypertube Hazardous Weather Hypervolume Prediction of 4-D Intersection* YN Occurrence of 4-D Intersection* YHitMiss N False Alarm Correct Rejection *Between Protected Aircraft Hypertube and Hazardous Weather Hypervolume CONTINGENCY MATRIX

11. MIT ICAT MIT ICAT Traditional Forecast Verification Methodology for  Hours Forecast Area Actual Adverse Weather Miss Hit False Alarm Correct Rejection Scores Based on Contingencies Critical Success Index73% Signal Detection Theory Hit Rate89% Signal Detection Theory False Alarm Rate17% Mean Square Error14%

12. MIT ICAT MIT ICAT Sensitivity of 4-D Intersection Test to Timing (e.g., ETD) Forecast Estimated Departure Time (Hrs) 4-D Intersection between Aircraft Hypertube and Weather Hypervolume No Yes 0 4 Correct Detection (A) Correct Rejection False Alarm (A) Forecast Area Aircraft A Aircraft B Correct Detection (B) False Alarm (B) 2

13. MIT ICAT MIT ICAT Trajectory-Based Weather Forecasting Opportunity Route Availability Planning Tool (RAPT) with MIT Lincoln Lab  Uses convective weather forecast + airline schedule  Predicts clear/blocked/impacted status of departure routes  Essentially 4-D intersection test Key Questions  What interface to provide to ATC users?  How to manage/communicate uncertainty?  Weather growth/decay  Aircraft trajectory  EncounterTiming Analytical approach Understand and model uncertainty in situation dynamics Characterize the geometry/kinematics of encounter Identify  Implications for available decisions  Opportunities for support the identification of options Provide recommendations for RAPT team & other tool development efforts

14. MIT ICAT MIT ICAT Summary of Analysis Relative Aircraft Track Angle Upstream: Aircraft trajectory toward boundary line Downstream: Aircraft trajectory away from boundary line Aircraft-Weather Encounter Situation Approaching weather boundary Receding weather boundary Error (Trajectory-Based) False alarms (FA)  Adverse weather is forecast to impact aircraft trajectory but does not Missed detections (MD)  Adverse weather is not forecast not to impact aircraft trajectory but does 8 Scenarios Potential causes Characterization Implications Adjustment options Relative Aircraft Track Angle Errors Aircraft-Weather Encounter Situation Upstream aircraft Downstream aircraft Approaching weather Receding weather

15. MIT ICAT MIT ICAT Example Worse Case Missed detection of approaching weather boundary for upstream aircraft trajectory Potential causes Front velocity underestimation Initiation/growth underestimation Aircraft velocity overestimation Characterization Transition to reduced capacity along route Associated with too late scenario Aircraft launch decision made in stochastic regime of weather representation Implications - Strategic Aircraft already in flight may have to divert Aircraft not already launched may have to delay departure for a long time Adjustment options - Tactical ATC: Identify aircraft trajectories around or above front Pilots: Identify gaps in front line Adjustment support ATC: Pre-negotiation between controllers for deviating aircraft flows Pilots: Support weather information update

16. MIT ICAT MIT ICAT Conclusions Weather decision-support requires addressing users’ time- dependent information needs Planning tasks Weather representation Under deterministic regime, trajectory-based weather forecasting has the potential to help support key decisions Matches users’ perspective and information needs for dynamics assessment  Pilots  ATC Aircraft-weather encounter analysis points to key risk adjustment strategies

17. MIT ICAT MIT ICAT

18. MIT ICAT MIT ICAT

19. MIT ICAT MIT ICAT Backup

20. MIT ICAT MIT ICAT Motivation Pilots’ information needs change over time Decision- Maker Situation Dynamics Information T t  Time of Information Applicability Time of Situation Dynamics Time of Planning Weather is chaotic Forecast accuracy decreases with forecast horizon Information is not sampled and provided continuously Pilots’ functions vary over time Human Decision-Maker Controlled System Environment Information t T2T2 11 22 T1T1 Sensors Displays Actuators

21. MIT ICAT MIT ICAT Uncertainty Growth with Forecast Horizon http://www.msc-smc.ec.gc.ca/cmc/verification/performance/index_e.html RMS Error in Geopotential Height Error Forecast Time Stochastic Deterministic Constant E(t) Weather representation based on observation over a time period where conditions do not significantly change Weather representation at time in future beyond “predictability limit” Weather representation based on deterministic forecast of “acceptable” accuracy

22. MIT ICAT MIT ICAT Predictability Horizon Linked to Weather Phenomenon Lifetime.01.1 1 10 100 1,000 10,000 1 sec. 1 min.1 hour1 day1 week Gust Building Wake Dust Devil Thermal CAT Shearing Gravity Wave Cumulus Downburst Rotor Lee Wave Cumulonimbus Severe Thunderstorm Gust Front Land/Sea Breeze Chinook Squall Line Hurricane Front Cyclone Jet Stream Monsoon Planetary Circulation Reference: P. F. Lester, Turbulence, A New Perspective for Pilots Horizontal Dimension (n.m.) Lifetime Microscale Mesoscale Macroscale

23. MIT ICAT MIT ICAT Matrix of Temporal Regimes of Cognitive Processes Temporal Regimes of Flight Planning Strategic Tactical Reactive  t t0 Temporal Regimes of Weather Representation Historical Constant Deterministic Stochastic 00 t0: time of information use  0: time of information production NOW re-routing around convective weather

24. MIT ICAT MIT ICAT Angle between weather front line and aircraft track  VFVF VFVF Front velocity perpendicular to the front line  Aircraft trajectory segment Scenario Analysis Model VAVA VAVA Aircraft velocity Storm line

25. MIT ICAT MIT ICAT Trajectory-Based Forecast Errors Front Departure Error False AlarmFly Faster More options Take-off ASAP More options Missed Detection Hold in air More options shortly Hold on ground More options shortly Front Arrival Error False Alarm Take-off ASAP No other options shortly Fly faster No other options shortly Missed Detection Hold on ground No other options Divert No other options Away from Front Toward Front N E S W Away from Front Toward Front Front Arrival Errors Affect Origin Airport Front Departure Errors Affect Destination Airport Relative Aircraft Track Angle Front Departure Errors Affect Origin Airport Front Arrival Errors Affect Departure Airport Aircraft Departure Time 1 2 3 4 5 6 7 8 Stochastic Regime Departure of Front from Critical Point Arrival of Front at Critical Point

26. MIT ICAT MIT ICAT Trajectory-Based Forecast Errors Types and Implications N E S W N E S W Toward Front Away from Front Front Departure Error False Alarm Take-off ASAP More options Fly Faster More options Missed Detection Hold on ground More options shortly Hold in air More options shortly Front Arrival Error False AlarmFly faster No other options shortly Take-off ASAP No other options shortly Missed Detection Divert No other options Hold on ground No other options Toward FrontAway from Front Front Arrival Errors Affect Destination Airport Front Departure Errors Affect Origin Airport Aircraft Track Angle Front Departure Errors Affect Destination Airport Front Arrival Errors Affect Origin Airport Aircraft Departure Time Accurate forecast has more value for longer impact time of adverse weather on aircraft route! Stochastic Regime Arrival of Front at Critical Point Departure of Front from Critical Point

27. MIT ICAT MIT ICAT Predictability Horizon Linked to Weather Phenomenon Lifetime.01.1 1 10 100 1,000 10,000 1 sec. 1 min.1 hour1 day1 week Gust Building Wake Dust Devil Thermal CAT Shearing Gravity Wave Cumulus Downburst Rotor Lee Wave Cumulonimbus Severe Thunderstorm Gust Front Land/Sea Breeze Chinook Squall Line Hurricane Front Cyclone Jet Stream Monsoon Planetary Circulation Reference: P. F. Lester, Turbulence, A New Perspective for Pilots Horizontal Dimension (n.m.) Lifetime Microscale Mesoscale Macroscale Multi-Cell Convective Line

28. MIT ICAT MIT ICAT Key Results

29. MIT ICAT MIT ICAT Area-based forecast assessment (top) vs.Trajectory-based forecast assessment (bottom) MD CD FA CR 4- D Trajectory Missed Detection (MD) Correct Detection (CD) False Alarm (FA) Correct Rejection (CR) Observation Forecast

30. MIT ICAT MIT ICAT Aircraft Trajectory Assessment for ATC Applications Aircraft Tracks inside Providence Sector 24 Hours – 1 mile grid Courtesy of Jonathan Histon

31. MIT ICAT MIT ICAT Weather Hypervolume Assessment

32. MIT ICAT MIT ICAT Metrics of Interest

33. MIT ICAT MIT ICAT Along Weather-Track Errors N E S W Correct Detection False AlarmMissed Detection

34. MIT ICAT MIT ICAT Along Weather-Track Errors Correct Detection False AlarmMissed Detection

35. MIT ICAT MIT ICAT Across Weather-Track Errors N E S W Correct Detection False AlarmMissed Detection

36. MIT ICAT MIT ICAT Across Weather-Track Errors Correct Detection False AlarmMissed Detection

37. MIT ICAT MIT ICAT Sensitivity of Trajectory-Based to Area-Based Performance Assessment Departure Time (Hours) Trajectory Angle (Degrees from North) Along Weather-Track Errors Across Weather-Track Errors Trajectory-Based % POD/FAR 20% Area-Based FAR and POD 20% Area-Based FAR and POD

38. MIT ICAT MIT ICAT Sensitivity Analysis Summary of Results Trajectory-based MD and FA correspond to 4-D intersections between 4-D aircraft hypertubes and FA/ MD hypervolumes Except when there are 4-D intersections between aircraft and CD hypervolumes, since they correspond to CD How should one measure ratios of POD or FAR? For fixed 4-D trajectories over departure time intervals?  What time intervals are relevant? For 2 simple cases investigated, the trajectory-based scoring will either look equal or better than the area-based statistics over the full time interval  Many exceptions

39. MIT ICAT MIT ICAT Sensitivity Analysis Summary of Results Trajectory-based errors depend on the geometry/kinematics Along weather-track errors are observed  At departure times when the weather is overhead the critical point  FAR is worse for certain time intervals  E.g., for trajectories with relative velocity (Va-Vw) perpendicular to the weather track  Linear relationship to timing error Across weather-track errors are observed  For trajectories that intersect the FA and MD hypervolumes FAR and POD  Highly dependent on the FA and MD hypervolume trajectories w.r.t. critical point

40. MIT ICAT MIT ICAT Angle between weather front line and aircraft track  VFVF VFVF Front velocity perpendicular to the front line  Aircraft trajectory segment Scenario Analysis Model VAVA VAVA Aircraft velocity

41. MIT ICAT MIT ICAT Correct Detection Forecast Error Angle between weather front line and aircraft track  VFVF VFVF Front velocity perpendicular to the front line  Aircraft trajectory segment Scenario Analysis VAVA VAVA Aircraft velocity

42. MIT ICAT MIT ICAT Summary & Implications Types of errors Front arrival  Marks transition to “blocked trajectory” state Front departure  Marks transition to “clear trajectory” state False Alarms  Missed opportunity if tactical response not supported Missed detections  Can be very disruptive  When related to front arrival errors, mean long ground hold or even diversion. Error type comparison Front arrival errors worse than front departure errors Missed detection worse than false alarm Worse when affecting destination airport  Stochastic regime  Affected by:  Front arrival errors when flying toward front  Front departure errors when flying away from front Value of finding “pores”/gaps: Greatest for front arrival forecasts Greatest for destination airports of aircraft

43. MIT ICAT MIT ICAT Opportunity for Applications

44. MIT ICAT MIT ICAT Methodology Cognitive Analysis of Pilots Decisions Feedback control loop Decision-making models Task analysis Temporal Representation Framework Deterministic Regime Stochastic Regime  Beyond the “deterministic predictability limit”  Importance of “options”  How to support “options” identification in planning?

45. MIT ICAT MIT ICAT

46. MIT ICAT MIT ICAT Uneventful Flight Hazardous Encounter with Wx Loss of Flight Opportunity Chance of Uneventful Flight Chance of Hazardous Encounter with Wx Go No-Go Gain Outcome Loss Outcome Sure Outcome Chance of Gain Chance of Loss Risk-Tolerant Action Sure Action The “gain” option identification is important in potentially hazardous weather If pilots frame the “no-go” decision as a loss, they will opt for the risk-tolerant “go” decision

47. MIT ICAT MIT ICAT Gain Outcome Loss Outcome Sure Outcome Chance of Gain Chance of Loss Risk-Tolerant Action Sure Action The “gain” option identification is important in potentially hazardous weather

48. MIT ICAT MIT ICAT Implications The “options” identification is a key element of risk assessment Supporting the identification of “options” explicitely may add value to the decision-support information

49. MIT ICAT MIT ICAT Conclusions How to improve time-varying decision-support Improve the “deterministic predictability limit” Understand how the deterministic regime supports tactical vs. strategic planning Understand how to support decision-making under the deterministic and stochastic regimes Under deterministic representation Decision-makers wish to have information that supports their assessment of the situation dynamics Under the stochastic representation Decision-makers wish to have information that supports their assessment of the availability of options

50. MIT ICAT MIT ICAT Decision- Maker Situation Dynamics Information T t  Time of Information Applicability Time of Situation Dynamics Time of Planning

51. MIT ICAT MIT ICAT Back Up Slides