Presentation on theme: "1 EUROCONTROL CARE - ATM Innovative Concepts Public Dissemination Forum EUROCONTROL HQ, Brussels, March 2004 Presenting Uncertainty to Controllers and."— Presentation transcript:
1 EUROCONTROL CARE - ATM Innovative Concepts Public Dissemination Forum EUROCONTROL HQ, Brussels, March 2004 Presenting Uncertainty to Controllers and Pilots David Nicholls, RM Consultants Ltd, UK Patrizia Marti and Valentina Barsotti, Deep Blue s.r.l., Italy risk management consultants NEW LOGO
2 Uncertainty in ATM If there were no uncertainty, would we need controllers and pilots …?
3 OUTLINE 1. Background 2. Objectives, Scope and Methods 3. Uncertainty in Conflict Detection 4 Uncertainty in Multi Sector Planning 5. Transfer to Other Contexts 6. Lessons Learned
4 1. BACKGROUND
5 Preliminary study (2001) RMC, University of Glasgow, pilot & HF consultants Where is there uncertainty in aviation? How do stakeholders think about and manage uncertainty? Where might there be benefit in presentation? What presentational forms are already used? Proposals for main CARE study
6 Types of uncertainty Variability Inaccuracy of measurement, prediction, display Missing data, lack of knowledge, incomplete understanding Information that is ambiguous or hard to interpret Subjective uncertainty - degree of belief or trust
7 ? ? ? ? Some aviation examples When will we get start clearance? Accuracy of TCAS Was that a real GPWS? Whats it doing now? Is the tailwind really only 10 kts? Land or go-around? Why are they sending me this way? How long will it take to climb? Flight Plan consistency Did he really understand? Shift handover in a hurry How big is the storm?
8 we dont have uncertainty in ATM we can only work with information that is certain Current approach to uncertainty A culture of eliminating or minimising uncertainty: more accurate and reliable tools standard operating procedures controllers/ pilots see their role as to reduce uncertainty
9 Why do more? Pilots and controllers base some decisions on incomplete or uncertain information, especially where prediction is involved New tools tend to increase the use of, and reliance on, predicted information Even where better accuracy/ reliability compensates for greater reliance, could we do even better by presenting uncertainty?
10 State of the art applicable guidance for ATM general psychology of uncertainty and decision making experimental studies for specific aviation tasks and tools Number of references on presenting uncertainty Generic Specific
11 Potential advantages more comprehensive picture for decision-making better indication that system is straying outside normal limits improved situation awareness, attention, enjoyment - an antidote to the problems of automation? Potential disadvantages increased display clutter increased cognitive load increased overall uncertainty in system and potential for confusion, because there is more scope for individual judgment Some advantages/ disadvantages of presenting uncertainty
12 2. OBJECTIVES, SCOPE AND METHODS
13 Objectives Develop, using case studies, principles for deciding what information about uncertainty is useful, how & to whom it should be presented Intended benefits: –practical guidance for ATM in general –contribute to the development of HMIs for case study tools
14 Case studies Conflict prediction (TP/ MTCD/ CORA) TLS= Tactical Load Smoother MSP= Multi-Sector Planner TP = Trajectory Prediction MTCD = Medium Term Conflict Detection CORA = Conflict Resolution Assistant Multi Sector Planning (TLS) The case studies relate to en-route ATC - not necessarily the most critical uncertainties in ATM Selected as examples of interest at the time, but lessons learned may be generalised
15 Scope of study Develop and evaluate prototypes to a level that allows an informed decision on more formal development (e.g. full simulation) Presentation, not reduction at source Information about uncertainty, not uncertain information Only uncertainties that are to some extent predictable - i.e. bounded and foreseeable Not concerned with verbal uncertainty
16 Overview of methods Multi-Sector Planning new concept - free to explore radical options development based on CREA! method - extensive user involvement new roles, tools and ways of working evaluation from animated storyboards Conflict Prediction existing tools, close to operational use traditional, problem- solving design approach add-ons to existing HMIs, exploring issues around data, algorithms evaluation in ERC Human Factors lab
17 Project organisation EUROCONTROL HQ RM Consultants Ltd Deep Blue s.r.l EUROCONTROL Research Centre Graffica Controller input: ENAV, CANAC, NATS
18 3. UNCERTAINTY IN CONFLICT PREDICTION (TP/ MTCD/ CORA)
19 Development/ evaluation method for conflict prediction Develop algorithms (simple distributions with real or subjective data Identify suitable tools and displays: VAW – uncertainty in climb/ descent PPD – uncertainty in conflict time and distance Identify suitable tools and displays: VAW – uncertainty in climb/ descent PPD – uncertainty in conflict time and distance Implement in eDEP platform Small-scale evaluation in ERC Human Factors lab analysis of MTCD/ TP/ CORA documents, simulation reports, discussions with designers and users workshop at ERC to brainstorm and short-list tools and sketch displays
20 Uncertainty in climb/ descent Time of initiating climb/ descent: triangular distribution Rate of climb/ descent: triangular distribution Variability about nominal climb/ descent: normal distribution, growing then decaying with time Smoothing and truncation
22 Uncertainty in conflict time and distance Along-track error: normal distribution, linear growth with time Cross-track error: uniform distribution, constant with time From these the MTCD algorithms calculate confidence bounds on when the CPA will occur and what the separation will be t = 1 t = 10 t = 1 t = 5
23 PPD without uncertainty TIME to CPA (minutes) DISTANCE (nm) at CPA User might decide to resolve conflict 3 before conflict
24 PPD with uncertainty TIME to CPA (minutes) DISTANCE (nm) at CPA certain and close 3 1 a long way off and uncertain a long way off but more certain 2 User might decide to resolve conflict 2 before conflict 3. Greater uncertainty = less urgent, in this context Ellipses show 95% confidence on time and distance Colour saturation increases as bubble shrinks, to counter the impression that big = important 0105
25 Experiment at ERC Two groups of 4 controllers, two days each Controllers worked both PC and TC roles No pilots, no R/T High and low traffic samples Uncertainty displays selected ON or OFF in each run Subjective measures – questionnaires, debriefings NASA TLX workload Initial results available
26 Questionnaire results Majority of negative views, except with regard to workload and colours of PPD More found PPD useful than VAW Number of controllers responding YesNo PPD increases SA? helpful? increases workload? would like one? matches perception of u/c VAW YesNo
27 TLX workload TLX - Mean and Standard Deviation ON OFF For high traffic, TLX* appears lower when uncertainty is displayed * TLX is a self-assessed, post- run measure (range 0-100) based on scores for mental and temporal demand, effort, performance and frustration. High traffic Low traffic 60 Uncertainty Display 20
28 Comments on controller views About half of the controllers were unfamiliar with MTCD/ PPD. Often comments (and scores?) related to these tools in general, not to the uncertainty displays specifically There were few occasions in which a conflict was seen against the VAW uncertainty area, and hence not many opportunities to use this display too many windows
29 Controller suggestions Better integration of the 3/ 4 dimensions Better integration with the whole-traffic picture - separate windows dont help build the picture and distract from or clutter the main radar view Younger controllers might like it Uncertainty displays could be useful (with training) for: –longer term planning roles e.g. MSP, –ground control and arrival/ departure planning (especially in a Collaborative Decision Making environment or with DMAN/AMAN)
30 4. UNCERTAINTY IN MULTI-SECTOR PLANNING
31 Deep Blue presentation ….
32 5. TRANSFER TO OTHER CONTEXTS
33 Transferability Case studies were for en-route ATC, and solutions must be context-specific But uncertainty presentation may be transferable to other contexts - especially where users have long look-ahead times and/ or major uncertainties in the information they use …..
34 Flight deck Pilots may benefit where there is: –increased use of predictive tools –full or partial self-separation (Free Flight) - pilots taking more responsibility for separation may need a shared appreciation of uncertainty with the controller –…..? Flight deck environment differs from control room: –much of pilots work is objective-oriented and immediate –main concern is with own aircraft, not with many –physical space for new displays is limited –…..
35 Arrival/ departure management ground & tower controllers and airlines involved in arrival/ departure management Uncertainty in ETA at KOKSI 0 KOKSI AirborneInboundTurn-roundTaxi On standPax/ Bags loadedPush backEngine start At runway hold Take off
36 6. LESSONS LEARNED
37 What to present Culture of eliminating uncertainty is deeply ingrained, but ….. Results of MSP study, particular, indicate potential benefits of presentation in some contexts Deciding, a priori, what specific uncertainty information will be useful is difficult
38 Who needs uncertainty information? Users with time to analyse decisions, and/or major uncertainties in information, such as: –long term planners (e.g MSP) –ground & tower controllers/ airlines? –pilots in Free Flight? Less useful for those making immediate decisions from short term information
39 Presentation means … Presentation is not limited to explicit display. It can include: making the uncertainty evident showing how the situation may develop - alternative futures providing what-if interactivity prompting the user to consider the certainty of their own judgment
40 Design rules - 1 provide what-if interaction give information about causes - whether within or outside user control visualise time - distinguish current and future indicate where tools / data are unreliable or incomplete Solutions must be scenario-specific, but some rules may have wider application:
41 Design rules - 2 present uncertainty on demand integrate with main traffic picture displays support correct inferences from new data (Bayesian updating, avoiding cognitive biases of anchoring/ recency) ensure a clear understanding of uncertainty but dont be too mathematical ….
42 Development process A CREA!-based process has been developed and demonstrated It uses real-life scenarios to introduce the idea of uncertainty and scenario-based, iterative, user-centered development. It appears more likely to lead to acceptable solutions than traditional problem-solving The CREA! method has been found to be scaleable, adaptable and effective