2. Episode 3 / CAATS II joint dissemination event Lessons Learnt Episode 3 - CAATS II Final Dissemination Event Philippe Leplae EUROCONTROL Episode 3 Brussels, 13 & 14 Oct 2009

3. Episode 3 - CAATS II Final Dissemination Event 2 Content Project objectives Lessons learnt method Lessons learnt in:  management  validation  concept clarification  result integration  validation techniques Conclusion

4. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 3 Project objectives Provide detail on key concept elements in SESAR Initial operability and performance assessment Initial technical impact assessment Analysis of available tools for SESAR validation Report on validation methodology …… Hence specific emphasis on lessons learnt

5. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 4 Lessons learnt method 2 workshops, Madrid and Brétigny 9 different sessions on 5 main topics involving more than 30 participants  Validation strategy and project DOW  Project management  Concept clarification  Integration and Consolidation  EP3 validation techniques and levels of maturity Most project members also working in SESAR Strong desire to provide feedback to SESAR

6. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 5 High level lessons learnt System view should drive validation activities  Need to adapt DOW and management to required flexibility  Need time to develop system driven project  We have developed artefacts to drive it Many different techniques tried on Episode 3  Suited to low maturity level concept elements  Guidelines on when and how to use it are produced Importance of support tools  Good strategy useless without tools to support it  Tools also require investment in training  This provides harmonized working methods, key to make distributed project work

7. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 6 Lessons learnt in management Our challenges:  Wide range of disseminated partners  Crossed responsibilities and thinly spread effort  Innovative tasks  High quality requirements  Tight deadlines Our answers:  Development of tools for distributed work  Training on the tools, and specific delivery process  A lot of co-ordination activities Follow-up actions:  Build more flexibility in the programme (DOW and contract)

8. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 7 Lessons learnt in validation Our challenges:  Maturity lower than expected at programme onset  E-OCVM V2 not well adapted to concept wide validation  Lack of E-OCVM expertise inside team  Wide range of results Our answers:  We built a framework with guidelines, principles, templates.  Developed assumptions management  Performance framework and DOD’s provided at start  Guidance provided through system WP and co-ordination cell  Thorough review of experimental plans Open issue:  Maturity assessment at start of each validation activity

9. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 8 Lessons learnt in concept clarification Our challenges:  Multiple sources from SESAR (CONOPS, OI’s,etc…)  Lack of details for validation  Multiple sources to update DOD’s Our solutions:  Propose a structure, around process model and DOD’s  Develop different modes of descriptions - DOD’s - Operational scenarios - Story boards  Develop collaborative working methods to update documents Follow-up:  Establish a recognised authority to select among options  Use hot topics to drive validation

10. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 9 Lessons learnt in result integration Our challenges:  Exercises not providing consistent results  Different granularity of results  Various degree of validity of quantitative data Our solutions:  Influence diagrams help understanding benefit mechanism and identify where performance is improved  Influence model has been developed ready to be fed with results  Management of assumptions to interpret results For subsequent coverage:  Steer validation activities from the performance framework  Use of models developed in EP3 to provide more quantitative data to performance model

11. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 10 Lessons learnt in validation techniques Successful used of different techniques  Prototyping  Macro-modelling  Various sorts or gaming  Expert groups Work packages integrated different techniques  Combine results of very different nature  Various sequences of activities tried  Expert groups key to success Benefits of industry assessment:  “Reality check”  Should run as a complementary process to operational validation

12. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 11 Validation techniques: Expert Groups Lessons learnt  Useful at all maturity levels  Need good facilitation techniques by somebody with OPS expertise  Need current operation experts + concept experts  Storyboards good method to structure EG debates Our input to SESAR  Storyboards technique  Various facilitation techniques  Hot topics  Guidelines document

13. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 12 Validation techniques: Gaming Lessons learnt  To be associated with Expert Groups and modelling  Suited to clarify roles and responsibilities for all maturity levels  More flexible than prototyping as not in real time (interruptions feasible)  Progression from paper-based gaming to platform gaming Our input to SESAR  Different platforms - CHILL - DARTIS - ACCES platform  Paper based gaming and associated web based tool  Guidelines document

14. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 13 Validation techniques: Prototyping Lessons learnt:  Not a cheap RTS!  Allowed to fail  Concept clarification-no quantitative data  Successive runs with same team of OPS staff Our input to SESAR  First results on: - PRNAV, CDA in descent phase. - Acceptability of 4D trajectory to controllers  Guidelines document

15. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 14 Validation techniques: Modelling Lessons learnt:  Use of process modelling and network modelling  Process modelling complement gaming  Need to work on precise concept elements  Macro-modelling Our input to SESAR:  3 innovative tools developed in EP3 - One abandoned (unsatisfactory results) - One providing results when applied to mature concept elements - One promising for network wide assessments  Guidelines document

16. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 15 Conclusion Episode 3 is the first ATM-wide concept validation programme We had to develop tools and methods adapted to this new challenge Another 2 months to complete our work (WP and final report) We provide to SESAR:  New tools  New methods  An integrated approach  First validation results…  See www.episode3.aerowww.episode3.aero And our most precious output:  A team with the expertise and concrete experience already working on the SESAR projects:

17. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 16

18. Thank you