Track 2 Summary Metrics in Validation One Sky for Europe EUROCONTROL

Participants from (alphabetically) –CENA –DLR –EUROCONTROL (EEC, HQ) –FAA (HQ, CAMI) –MITRE –NASA Ames –NATS –NLR –San Jose State University –Volpe Center

Areas Strategic Top/Down and Bottom Up - 3 papers Operational Context - 3 Human Performance - 2 Analytic - 1

Question 1 - Can operationally relevant metrics also be used in simulation environments to predict the benefits of the concept? Yes, but … Simulation results are not always extensible to operations Airport arrival rate, throughput, etc. can be used Operational metrics should be explicitly defined for simulation environments Key operational measures need to be supported by other measures (cost, technical, human performance, etc.) Further detailed metrics are needed to assess constraints and stability of concepts When early in development, operational metrics have limited interpretation Risk buffer shrinks (that is, confidence increases) with consistent trends in results

Question 2 - Can these metrics be generated by our simulation and validation tools; what level of confidence can they give? It depends … Things change in transition to the field Need early validation results Converging results from multiple measures –Contributions of other lower level measures Different levels of confidence for short versus longer term solutions Distinctions between quantitative and qualitative data Need clarity in definition of metrics –Iterative identification of measures targeting issues/questions

Question 2 Continued - Can these metrics be generated by our simulation and validation tools; what level of confidence can they give? Confidence is proportional to confidence in validation tools, assumptions, and methods (in reference to the banana) Need to integrate top-down approach in deriving metrics from requirements (operational, technical) and bottom-up approach What level of confidence is needed? Expected? How should confidence be expressed?

Question 3 - To what extent are metrics concept dependent and allow comparisons across validations? For the most part … Many metrics are concept independent –Tailoring to a concept may reduce extension to other concepts In particular need measures of decision support tool interoperability –Tailor to level of tool maturity Consider definition of scenarios and research processes Consistent use of VDR: –Prevents re-invention –Identifies where information is absent –Supports increased confidence

Questions and Recommendations Questions Areas for future collaboration? –Priorities? Areas for improvement? Recommendations Improve metrics to have more understanding of their limitations in order to have confidence in results Bridge gap between human performance metrics and system metrics Have a clearly defined concept and a process for defining a quality test including necessary data and scenarios