1. 1. KPAs, KPIs, and metrics
Performance measurement in a world of targets and trade-offs

2. KPAs & metrics
Key Performance Areas represent some qualitative information about what is interesting in a system.
Example: ‘Safety’
KPA is vague, almost by definition, multidimensional, and subjective.
KPA comes from a paradigm and a moral => Political by nature. Vs
Metrics are observables
Example: ‘Number of accidents’
A metric is unambiguous, usually unidimensional, and objective.
Metrics do not come from a symbolic social representation but from physical observations.
Performance measurement in a world of targets and trade-offs

3. KPAs & metrics KPA Metric 1 Metric 2 Metric 3 Metric 4 Metric 5
Performance measurement in a world of targets and trade-offs

4. KPAs & metrics KPA A Metric 1 Metric 2 KPA B Metric 3 KPA C Metric 4
KPA D
Performance measurement in a world of targets and trade-offs

5. Making the bridge: the indicator
Different steps for choosing metrics:
Top-down: break down the KPA in ‘atomic’ issues:
Example: ‘safety?’ => deaths, injuries, psychological, physical, etc
Bottom-up: see what you can measure
Go through the measurable metrics and relate them to atomic issues.
Done? No!
4. Define an acceptable level
Key Performance Indicator: a metric with a threshold.
KPIs are often thought to be objective. They are not!
Performance measurement in a world of targets and trade-offs

6. Importance of statistics
Not measuring one thing, but many instances of the same metric => One value at the end?
Humans not hard-wired to deal with statistics.
Average measures?
Example: if the average revenues of the airlines increase, most of the airlines gain more money?
Quantile?
Example: 4HD2D is supposed to be reached by 90% of the population. What about the 10%?
=> Statistics need to be embedded directly into (some) KPIs.
Example: average delay should not be provided without interval corresponding e.g. to first and last quartile.
Sampling for rare events?
Performance measurement in a world of targets and trade-offs

7. 2. Targets and the future of the European ATM system
Performance measurement in a world of targets and trade-offs

8. Targets
A target is the threshold associated to a metric => indicator
It is set in order to drive forward some procedures, and is thus optimistic
A target is an educated guess of expert in the fields based on little quantitative data.
If the target is too high, the target is unattainable and is considered irrelevant. If the target is too low, it is useless to drive innovation forward.
Targets need to reevaluated based on bottom-up data: the carrot needs to always be one step ahead of the donkey.
Set a target for a general domain: drives the innovation in the subdomains and forces to select the best competing solutions.
Set a target for a specific sub-domain: drives the design process of a solution. Capability-based.
Set a target at a specific time horizon: drives the implementation of the potential solutions. Time-based.
Performance measurement in a world of targets and trade-offs

9. What does the future look like?
Vista and DATASET2050 models will run on several scenarios based on three time scales:
‘Current’ (2020): Are will able to understand the relation between KPIs now?
Mid-term (2035): If some technology currently under development its fully implemented, what will be its impact?
Long-term (2050): what kind of changes could happen then, and if they happen, what are the consequences?
Performance measurement in a world of targets and trade-offs

10. Guessing the future with targets
Two (fairly) independent questions:
What will the operational environment look like? => Difficult
What are the consequences of these possible operational environments? => Slightly less difficult
Capabilities based on SESAR targets for capabilities (e.g. Master plan Ed2)
Time horizons based on SESAR time-based targets (e.g. Master plan Ed3)
Performance measurement in a world of targets and trade-offs

11. Old SESAR targets: step 1, step 2, step 3
Master plan Ed 2 gives very detailed target for three ‘steps’:
Step 1: time-based operations:
time prioritisation for arrivals at airports is initiated;
datalink is widely used;
initial trajectory-based operations are deployed
Step 2: trajectory-based operations:
partners optimise “business and mission trajectories” through common 4D trajectory information
users define priorities in the network
Step 3: performance based operations:
high‐performance, integrated, network-centric, collaborative and seamless air/ground ATM system
The steps are time-agnostic
Performance measurement in a world of targets and trade-offs

12. Example: Airspace Capacity
Why this shape?
Performance measurement in a world of targets and trade-offs

13. Example: Airport Capacity
Sublinear: low hanging fruits first, then refinements.
Superlinear: first improvements help further ones.
Performance measurement in a world of targets and trade-offs

14. Example: Predictability
Predictability is one of the major SESAR (and ICAO) KPIs: variance of departure times.
Why constant reduction? Why not constant factor?
Performance measurement in a world of targets and trade-offs

15. Time-horizon: mapping
1st phase: finding out which step is roughly consistent with the latest time-based targets.
Targets roughly consistent with step 3
Performance measurement in a world of targets and trade-offs

16. Time-horizon: Extra/interpolation
2nd phase: Extrapolating (and interpolating). If step 3 is reached by 2035, what happens in 2025, 2040, 2050?
Linear extrapolation: negative variance??
Performance measurement in a world of targets and trade-offs

17. Time-horizon: Extra/interpolation
2nd phase: Extrapolating (and interpolating). If step 3 is reached by 2035, what happens in 2025, 2040, 2050?
Reduction by the last factor?
Performance measurement in a world of targets and trade-offs

18. Time-horizon: Extra/interpolation
2nd phase: Extrapolating (and interpolating). If step 3 is reached by 2035, what happens in 2025, 2040, 2050?
Reaching the long-term target by 2050 with a constant factor?
Performance measurement in a world of targets and trade-offs

19. Time-horizon: Extra/interpolation
2nd phase: Extrapolating (and interpolating). If step 3 is reached by 2035, what happens in 2025, 2040, 2050?
Decreasing by a constant factor given by the first reduction
Performance measurement in a world of targets and trade-offs

20. Time-horizon: Validation exercises
3rd phase: adjusting with validation exercises
Performance measurement in a world of targets and trade-offs

21. Time-horizon: disaggregation
4th phase: estimations of the target for sub-domains (SESAR SubPackages)
Performance measurement in a world of targets and trade-offs

22. Time-horizon: optimism and pessimism
5th phase: build optimistic and pessimistic cases
Performance measurement in a world of targets and trade-offs

23. 3. Further exploration of trade-offs
Performance measurement in a world of targets and trade-offs

24. Trade-offs
Performance measurement in a world of targets and trade-offs

25. How to study KPIs relationships?
First method: through modelling
=> Deep understanding, possibility of exploring parameters
Problem: assumptions of model realistic?
Performance measurement in a world of targets and trade-offs

26. How to study KPIs relationships?
Second method: through statistics
=> does not rely on any assumption
No understanding of ‘why’, needs lots of data, sensible to changing conditions.
Performance measurement in a world of targets and trade-offs

27. Do we need more KPIs?
The multiplication of metrics is to some extent useless => minimal description of the system?
Performance measurement in a world of targets and trade-offs

28. 4. Questions for discussion
How political is a target? How much should it be data-driven?
Are capability-based targets reliable?
Is the impact assessment of a solution really easier than forecasting its uptake?
Is the inter/extrapolation reasonable? Can we do better (surely we can)?
Are optimistic/pessimistic scenarios useful? Can we put some rough probability of realisation on them?
Are trade-offs compatible with targets?
Future challenges
have we missed anything out of our list?
what are the priorities?
do we need a trade-off tool for stakeholders and how would it work?
Next steps and best collaborative approach?
Performance measurement in a world of targets and trade-offs

29. Thanks for your attention!