Presentation on theme: "Presenting Uncertainty to Controllers and Pilots"— Presentation transcript:
1Presenting Uncertainty to Controllers and Pilots EUROCONTROL CARE - ATM Innovative ConceptsPublic Dissemination ForumEUROCONTROL HQ, Brussels, March 2004Presenting Uncertainty to Controllers and PilotsDavid Nicholls, RM Consultants Ltd, UKPatrizia Marti and Valentina Barsotti, Deep Blue s.r.l., Italyrisk management consultantsNEW LOGO
2Uncertainty in ATMIf there were no uncertainty, would we need controllers and pilots …?
3OUTLINE 1. Background 2. Objectives, Scope and Methods 3. Uncertainty in Conflict Detection4 Uncertainty in Multi Sector Planning5. Transfer to Other Contexts6. Lessons Learned
5Proposals for main CARE study Preliminary study (2001)RMC, University of Glasgow, pilot & HF consultantsWhere is there uncertainty in aviation?How do stakeholders think about and manage uncertainty?Where might there be benefit in presentation?What presentational forms are already used?Proposals for main CARE study
6Types of uncertainty Variability Inaccuracy of measurement, prediction, displayMissing data, lack of knowledge, incomplete understandingInformation that is ambiguous or hard to interpretSubjective uncertainty - degree of belief or trust
7Some aviation examples When will we get start clearance?Accuracy of TCASWas that a real GPWS?What’s it doing now?Is the tailwind really only 10 kts?Land or go-around?Why are they sending me this way?How long will it take to climb?Flight Plan consistencyDid he really understand?Shift handover in a hurryHow big is the storm???
8Current approach to uncertainty A culture of eliminating or minimising uncertainty:more accurate and reliable toolsstandard operating procedurescontrollers/ pilots see their role as to reduce uncertaintywe don’t have uncertainty in ATMwe can only work with information that is certain
9Why do more?Pilots and controllers base some decisions on incomplete or uncertain information, especially where prediction is involvedNew tools tend to increase the use of, and reliance on, predicted informationEven where better accuracy/ reliability compensates for greater reliance, could we do even better by presenting uncertainty?
10applicable guidance for ATM State of the artNumber of references on presenting uncertaintygeneral psychology of uncertainty and decision makingexperimental studies for specific aviation tasks and toolsapplicable guidance for ATMGenericSpecific
11Some advantages/ disadvantages of presenting uncertainty Potential advantagesmore comprehensive picture for decision-makingbetter indication that system is straying outside normal limitsimproved situation awareness, attention, enjoyment - an antidote to the problems of automation?Potential disadvantagesincreased display clutterincreased cognitive loadincreased overall uncertainty in system and potential for confusion, because there is more scope for individual judgment
13ObjectivesDevelop, using case studies, principles for deciding what information about uncertainty is useful, how & to whom it should be presentedIntended benefits:practical guidance for ATM in generalcontribute to the development of HMIs for case study tools
14Case studies Conflict prediction (TP/ MTCD/ CORA) Multi Sector Planning (TLS)The case studies relate to en-route ATC - not necessarily the most critical uncertainties in ATMSelected as examples of interest at the time, but lessons learned may be generalisedTLS = Tactical Load SmootherMSP = Multi-Sector PlannerTP = Trajectory PredictionMTCD = Medium Term Conflict DetectionCORA = Conflict Resolution Assistant
15Scope of studyDevelop and evaluate prototypes to a level that allows an informed decision on more formal development (e.g. full simulation)Presentation, not reduction at sourceInformation about uncertainty, not uncertain informationOnly uncertainties that are to some extent predictable - i.e. bounded and foreseeableNot concerned with verbal uncertainty
16Multi-Sector Planning Overview of methodsConflict Predictionexisting tools, close to operational use‘traditional’, problem- solving design approachadd-ons to existing HMIs, exploring issues around data, algorithmsevaluation in ERC Human Factors labMulti-Sector Planningnew concept - free to explore radical optionsdevelopment based on CREA! method - extensive user involvementnew roles, tools and ways of workingevaluation from animated storyboards
17Project organisation EUROCONTROL HQ EUROCONTROL Research Centre RM Consultants LtdDeep Blue s.r.lGrafficaController input: ENAV, CANAC, NATS
183. UNCERTAINTY IN CONFLICT PREDICTION (TP/ MTCD/ CORA)
19Development/ evaluation method for conflict prediction Identify suitable tools and displays:VAW – uncertainty in climb/ descentPPD – uncertainty in conflict time and distanceanalysis of MTCD/ TP/ CORA documents, simulation reports, discussions with designers and usersworkshop at ERC to brainstorm and short-list tools and sketch displaysDevelop algorithms (simple distributions with real or subjective dataImplement in eDEP platformSmall-scale evaluation in ERC Human Factors lab
20Uncertainty in climb/ descent Time of initiating climb/ descent: triangular distributionRate of climb/ descent: triangular distributionVariability about nominal climb/ descent: normal distribution, growing then decaying with timeSmoothing and truncation
21Vertical Aid Window (VAW) Late, shallow descentEarly, steep descent
22Uncertainty in conflict time and distance Along-track error: normal distribution, linear growth with timeCross-track error: uniform distribution, constant with timeFrom these the MTCD algorithms calculate confidence bounds on when the CPA will occur and what the separation will bet = 1t = 1t = 5t = 10
23PPD without uncertainty 15210TIME to CPA (minutes)35User might decide to resolve conflict 3 before conflict 21510DISTANCE (nm) at CPA
24PPD with uncertainty Ellipses show 95% confidence on time and distance a long way off but more certain152a long way off and uncertain10TIME to CPA (minutes)3certain and close5User might decide to resolve conflict 2 before conflict 3. Greater uncertainty = less urgent, in this context.1510DISTANCE (nm) at CPAEllipses show 95% confidence on time and distanceColour saturation increases as bubble shrinks, to counter the impression that ‘big = important’
25Experiment at ERC Two groups of 4 controllers, two days each Controllers worked both PC and TC rolesNo pilots, no R/THigh and low traffic samplesUncertainty displays selected ON or OFF in each runSubjective measures – questionnaires, debriefings NASA TLX workloadInitial results available
26Questionnaire results Number of controllers respondingYesNoYesNohelpful?increases SA?increases workload?matches perception of u/cwould like one?VAWPPDMajority of negative views, except with regard to workload and colours of PPDMore found PPD useful than VAW
27TLX - Mean and Standard Deviation TLX workloadUncertainty DisplayFor high traffic, TLX* appears lower when uncertainty is displayed* TLX is a self-assessed, post-run measure (range 0-100) based on scores for mental and temporal demand, effort, performance and frustration.ONOFF7060High traffic50TLX - Mean and Standard Deviation40Low traffic3020
28Comments on controller views About half of the controllers were unfamiliar with MTCD/ PPD. Often comments (and scores?) related to these tools in general , not to the uncertainty displays specificallyThere were few occasions in which a conflict was seen against the VAW uncertainty area, and hence not many opportunities to use this display‘too many windows’
29Controller suggestions Better integration of the 3/ 4 dimensionsBetter integration with the whole-traffic picture - separate windows don’t help build the picture and distract from or clutter the main radar view‘Younger controllers might like it’Uncertainty displays could be useful (with training) for:longer term planning roles e.g. MSP,ground control and arrival/ departure planning (especially in a Collaborative Decision Making environment or with DMAN/AMAN)
33TransferabilityCase studies were for en-route ATC, and solutions must be context-specificBut uncertainty presentation may be transferable to other contexts - especially where users have long look-ahead times and/ or major uncertainties in the information they use …..
34Flight deck Pilots may benefit where there is: increased use of predictive toolsfull or partial self-separation (‘Free Flight’) - pilots taking more responsibility for separation may need a shared appreciation of uncertainty with the controller…..?Flight deck environment differs from control room:much of pilots’ work is objective-oriented and immediatemain concern is with own aircraft, not with manyphysical space for new displays is limited…..
35Arrival/ departure management ground & tower controllers and airlines involved in arrival/ departure managementUncertainty in ETA at KOKSIPax/ Bags loadedEngine startAt runway holdKOKSIPush backOn standTake offInboundTurn-roundTaxiAirborne
37What to presentCulture of eliminating uncertainty is deeply ingrained, but …..Results of MSP study, particular, indicate potential benefits of presentation in some contextsDeciding, a priori, what specific uncertainty information will be useful is difficult
38Who needs uncertainty information? Users with time to analyse decisions, and/or major uncertainties in information, such as:long term planners (e.g MSP)ground & tower controllers/ airlines?pilots in Free Flight?Less useful for those making immediate decisions from short term information
39Presentation means …‘Presentation’ is not limited to explicit display. Itcan include:making the uncertainty evidentshowing how the situation may develop - alternative futuresproviding ‘what-if’ interactivityprompting the user to consider the certainty of their own judgment
40Design ‘rules’ - 1Solutions must be scenario-specific, but some ‘rules’ may have wider application:provide ‘what-if’ interactiongive information about causes - whether within or outside user controlvisualise time - distinguish current and futureindicate where tools / data are unreliable or incomplete
41Design ‘rules’ - 2 …. present uncertainty on demand integrate with main traffic picture displayssupport correct inferences from new data (Bayesian updating, avoiding cognitive biases of anchoring/ recency)ensure a clear understanding of ‘uncertainty’ but don’t be too ‘mathematical’
42Development processA CREA!-based process has been developed and demonstratedIt uses real-life scenarios to introduce the idea of uncertainty and scenario-based, iterative, user-centered development.It appears more likely to lead to acceptable solutions than traditional ‘problem-solving’The CREA! method has been found to be scaleable, adaptable and effective