1. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 1 Episode 3 - Ex-Com Final Report Tools & Validation Dr. Ralph Leemüller R&D Division - DFS

2. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 2 Brussels, 13 & 14 Oct 2009 2 Overview Validation Methodology Innovative Validation Techniques and Tools Expert Groups Gaming Techniques Prototyping Sessions Modelling and Fast-time Simulations Combined Techniques Episode 3 - CAATS II Final Dissemination Event

3. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 3 3 Concept Validation Methodology Concept lifecycle (E-OCVM) Use of E-OCVM Episode 3 adressed Steps V1 – V3 All validations in accordance to methodology Validation strategy Validation plans – Conduction of Exercise – Report

4. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 4 What is an Expert Group Expert group working on a set of predetermined concepts in a structured manner Gain from group expertise specific professional profiles using both the individual skills and the synergy of the group More complete vision; Larger number of factors; More reliable The mission of an Expert Group in a validation project is providing: A reliable approach to real cases, even when assuming future situations; and Experience based judgement analysis for supporting the feasibility of the developments to be undertaken.

5. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 5 Using Expert Groups Pros Concepts development, refinement and clarification; Qualitative assessments; Prediction of potential issues regarding future situations; Support validation exercises design and development. Analysis of consistency of validation results. Cons Difficulty to produce quantitative results; Limited availability of the experts; Biased vision; Trend to keep things the way they are. Outputs: Operational details: refinement of DoDs and scenarios; Refinement of the approach of the validation exercises; Judgement of exercise results

6. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 6 Expert Group 6 Addressing the Concept Validation - Expert Groups Concept clarification and operational feasibility of processes Concept layered phases:  Expert Groups are suitable for the initial validation stage (concept definition and clarification) as well as to consider cross- functional issues, i.e. initial performance assessment and operational feasibility;  Pre-industrial development can be supported by experts. Concept layered phases:  Expert Groups are suitable for the initial validation stage (concept definition and clarification) as well as to consider cross- functional issues, i.e. initial performance assessment and operational feasibility;  Pre-industrial development can be supported by experts.

7. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 7 What is Gaming Human in the loop activity applicable to simulate decision making processes; Experts are ‘around a table’ playing different roles as they were living real processes (AOCs, Pilots, AMC, Network Manager,...); Every role has a particular scope, objectives and strategies during the process; Scenario running in real-time (or even slow time); Promoting non-routine thinking, thus enabling out-of-the-box thinking.

8. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 8 Using Gaming Pros Clarification of the concept; Qualitative assessment of processes feasibility: Detection of bottlenecks and shortcomings; Clarification on roles and responsibilities: Identification of the information flows among actors; Exploration on supporting tools requirements. Cons Mostly qualitative assessment; quantitative results from SW-Games Without an adequate guidance the Gaming session may turn into an Expert Group; Results obtained during the game are dependant on the participants (expertise) and platform limitations. Outputs: Refinement of Operational details (Information flows, roles & responsibilities);

9. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 9 9 Supporting means to play Three different gaming approaches:  Paper-based game;  Web-based game;  SW Platform-based: - DARTIS; ACCES; - CHILL; - PROMAS. Three different gaming approaches:  Paper-based game;  Web-based game;  SW Platform-based: - DARTIS; ACCES; - CHILL; - PROMAS.

10. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 10 Addressing the Concept Validation – Gaming Concept layered phases:  All phases can be covered by gaming (Planning and Execution Phases);  Suitable for clarification on components of the concept better than for global concept. Concept layered phases:  All phases can be covered by gaming (Planning and Execution Phases);  Suitable for clarification on components of the concept better than for global concept. Gaming Exercise Paper-basedPlatform-based Concept clarification and operational feasibility of processes

11. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 11 What is prototyping session Human in-the-loop experiments to build the ATM environment piece by piece and to address detailed issues (e.g. airspace, tools, roles). The focus is on operability: Real-time technique Small-scale and/or part-task Iterative in nature Gradually building and refining Examine the options, moving towards a clear goal

12. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 12 Using Prototyping Session Pros Flexibility Quick answers Suited to early concept assessment/maturation Reduced cost: small scale and simplified environment, to focus on specific issues Cons Short-time scales are the major limitation One week sessions Time between sessions Training time Examinination of the concept with multiple conditions Outputs: Assessment of operability and acceptability refinement/maturation of a defined set of concept elements Initial trends on some KPAs

13. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 13 Addressing the Concept Validation – Prototyping Session Concept layered phases:  Execution phases is covered by prototyping;  Suitable for feasibility on components of the concept better than for global concept. Concept layered phases:  Execution phases is covered by prototyping;  Suitable for feasibility on components of the concept better than for global concept. Proto- typing Operational feasibility

14. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 14 What is Modelling and Fast-time Simulation Using models of ATM systems Use a representation of elements of the ATM concept Applied when it is impractical to recreate the experimental conditions with human in the loop (large scope, long duration, wide area, …) Predict measures related to physical processes or to assess feasibility of the processes.

15. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 15 Using Modelling and FTS Pros Large scope in time or geographic area Short time of simulation run Reduced cost - targeted use of expert Reproducible Quantitative results on KPA’s (i.e. Capacity, Efficiency) Cons Model agents may introduce simplifications Analysis and interpretation requires an experienced expert Level of maturity of concept influence the accuracy of output Outputs: Assessment of quantitative results (KPA) refinement/maturation of a defined set of concept elements

16. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 16 Addressing the Concept Validation – Modelling and FTS Concept layered phases:  All phases can be covered by gaming (Planning and Execution Phases).  Macro model to extrapolate results on ECAC-wide applicability  refinement/maturation of a defined set of concept elements  Suitable for concept assessment on wide area and large coverage. Concept layered phases:  All phases can be covered by gaming (Planning and Execution Phases).  Macro model to extrapolate results on ECAC-wide applicability  refinement/maturation of a defined set of concept elements  Suitable for concept assessment on wide area and large coverage. Model & FTS ModellingFast-Time Concept clarification and Feasibility of large scope, long duration

17. Episode 3 E3-WP0-MWPM20091215-PPT-V0001-5 th ExCom Slides 17 Combining Techniques