1. 27 June 20031 Metrics and Performance Management Session Report Alison Hudgell

2. 27 June 20032 Introductory remarks Topic of increasing importance –high-level performance targets –demonstrate benefits of new developments Difficult in complex, multi-faceted system A real need for metrics that –reflect what we are trying to achieve –can measure our progress, year on year –can be understood by decision makers at high levels in all parts of the industry

3. 27 June 20033 Topics addressed & major results Metrics for classification –Sector Complexity Index from workload per flight, using TAAM [58] –Simple metrics, quick application to identify areas suitable for “super sectors” [34] –Spectral analysis of delays used to identify sets of similar airports [46] Categorisation identifies like sectors/areas/ airports that can be treated as a group –for applying analysis techniques –for approaches to operational improvement

4. 27 June 20034 Topics addressed & major results Examples... Sector categorisation by binary tree using different complexity factors [44] –Applied to improve estimates of declared capacity based on simple workload measure Sector “efficiency” from comparing declared capacity with complexity measure [10] –Critical but inefficient sectors could increase capacity by emulating “best in class”

5. 27 June 20035 Topics addressed & major results Sector complexity [7] [86] –Attempts to fuse factors from proposed complexity metrics, using statistical analysis, to give a “best” metric –Reduced metric better match to controller assessment of complexity than traffic count alone

6. 27 June 20036 Topics addressed & major results Statistical analysis of conflict clusters in simulated traffic [33] –Direct routes provide smaller & less complex clusters than structured routes –Uncertainty increased number and size of clusters –Uncertainty had more impact in structured than direct routes case

7. 27 June 20037 Topics addressed & major results Flight efficiency benefit pool estimation [23] –Consistency between studies estimating benefit pool for en-route flight time in US –Order of $50-100 or 1-2 mins per flight

8. 27 June 20038 Topics addressed & major results Performance assessment of ATM system development projects [92] –Generalised Activity Network approach captures reality of project lifecycles –Gives insight into which stages likely to overrun –Combines to estimate likely extent of total overrun –Compatibility with budget allocation processes?

9. 27 June 20039 Topics addressed & major results Need for standard metrics framework [28] –1500 metrics used for analysis of ATM systems –600 after eliminating duplicates –Impact on communication, comparability of studies –Standardisation of metrics required –Vertical structure is key: breakdown from goals -> objectives -> metrics and re-aggregation ›FAA/EUROCONTROL workshop –ref. Wim Post (EUROCONTROL)

10. 27 June 200310 Issues discussed What is the goal? –E.g. flight efficiency - compare flown route with direct route or airline requested route or...? Presented metrics/methodologies can only capture part of the picture –E.g. flight efficiency - consider en-route delay, but how add effect of ground delay? Data availability –Metrics must be pragmatic - if data not available, can’t use “ideal” metric

11. 27 June 200311 Issues discussed Appropriate scoping and approach –Deterministic or stochastic approaches –Accommodating uncertainty in forecasts –Linear vs non-linear models –Adaptive behaviour system complicates analysis –Degree of aggregation: dynamics (peaks) vs time- average

12. 27 June 200312 Progress & trends Reported work focuses on capacity, complexity and delay Metrics work for ATM is not mature Complexity work reaching a new phase –now trying to compare, assess and reduce to a standard set of factors –work needs to converge - how can we reach accepted measures of complexity?

13. 27 June 200313 Progress & trends Recent increase in data availability in US –but still a problem of lack of some types of data Sensitivity analyses are needed but not always conducted

14. 27 June 200314 Future directions? A real need for comprehensible metrics to support decision making Clarification of goals Standard, validated framework –agreement of terms –breakdown from goals -> objectives -> metrics and re-aggregation Trade-offs (capacity/delay/safety/efficiency) Safety metrics?