1. Institut für Technische Informatik Fakultät für Informatik Universität der Bundeswehr München ITIS e.V. Institut für Technik Intelligenter Systeme an der Universität der Bundeswehr München NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Email: axel.lehman@unibw.deaxel.lehman@unibw.de http://www.unibw.de/inf4/personen/prof/al/index_html Verification, Validation and Accreditation of Simulation Models and Applications Axel Lehmann

2. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Technology Trends  Rapid technological innovations  information and communication technologies  computer-assisted tools, e.g. for  Design & analysis support  Planing & decision support  Education & training  Increasing systems complexity & lifetime  embedded systems  networked components / systems  distributed systems  Increasing productivity & cost-benefit relations  Decreasing cycle times for system’s innovations

3. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Major Challenges  “Mastering” of system(s) complexity over lifetime w.r.t. multiple aspects/ goals? (safety, reliability, performance,…)  Approach: Goal-oriented, efficient, effective and dependable models!  “Mastering” of model(s) complexity !?  STRATEGIC approach: “Devide and Conquer”!  MODEL-ENGINEERING approach!!

4. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 M&S-Approach: “Divide and Conquer” improving EFFICENCY increasing CREDIBILITY Multi-Phase M&S process(es) (intermediate products) verification and validation correctness; suitability reusability; distributed execution M&S „components“

5. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Outline 1.M&S: A Multiple-Phase Development & Application 2.Improving M&S-Efficiency by Component-Based M&S 3.Increasing Credibility through M&S-Verification & Validation 4.Conclusion

6. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 1. M&S – A Multiple-Phase Development and Application Process  Analyses goals e.g.:  processing time per transaction  client/server utilization  queueing  How to approach this (complex) problem?  “Divide and Conquer”:  A Multiple-Phase M&S Process  Example:Effectivness and efficiency of a „Booking System“ (e.g. client-server architecture)

7. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 M&S – Development Team (Skills) Modeling Expertise Experimental Design and Analysis Domain Knowledge (HW-)SW- Expertise User Knowledge Problem Definition System Analysis Model Formalization Implementation Experimentation Programmer Modeller Domain Expert User Customer Project Manager (Contractor)

8. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Phases & Products in the M&S- Development Process Model Input Data Solution Techniques Modeling Method System Observations Experimental req.&constr. Technical req.&constr. Formal req.&constr. Conceptual req.&constr. Examination Aim Input: Problem Definition System Analysis Model Formalization Implementation Experimentation Phase: Model Documen- tation Objekt I n t e r a k t i o n Systemgrenze “Umwelt- objekt“ “Hauptobjekt” mit Modellattributen Communicative Conceptual Model, Project Objectives Structured Problem Description Formal Model Executable Model Results Intermediate Products:

9. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Example: „Booking System“ I/O device 1 CPU T 1 T n I/O device n Server Clients  Conceptual Model:

10. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Example: „Booking System“  Formal Model: peripheral devices

11. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Example: „Booking System“  Executable model: Performance measures  Little´s law: k = + t ; with: response time t, queueing time w, service rate µ t = w + ; µ 1 1      m service rate arrival rate  State probability p(k): k =  k p (k)  Utilization  ( m service stations):

12. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 2. Model “Component”-Based Approach:  Executable Model(s): e.g.  analytic solution  simulation reusable SW- “components“ (e.g. regarding the example „Booking System“)  Problem Description:  pragmatism/ goal specification  Conceptual Model:  structural & functional description of “components”  different levels of abstraction  Formal Model:  formal specification of “components” (  selected modeling paradigm(s))  hierarchical modeling approach (  decomposition into submodels/ “components”)

13. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Phases & Products in the M&S- Development Process t M&S- „components“: X X XX X Model Input Data Solution Techniques Modeling Method System Observations Experimental req.&constr. Technical req.&constr. Formal req.&constr. Conceptual req.&constr. Examination Aim Input: Model Documen- tation Objekt I n t e r a k i o n Systemgrenze “Umwelt- objekt“ “Hauptobjekt” mit Modellattributen Communicative Conceptual Model, Project Objectives Structured Problem Description Formal Model Executable Model Results Intermediate Products: Problem Definition System Analysis Model Formalization Implementation Experimentation Phase:

14. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Model („Component“) Specification Levels Model repository  Model Federation Level: „Black Boxes“ Library of submodels & communication infrastructure  Model Level: Autonomous, interoperable models Library of objects/methods (for interaction)  Submodel/ Object Level: Submodels/ Object structures  60und0mit :0 0)(wenn:),()()(. 111 1         sni sonst tlttzttrftl tls isiisiissiis is Program Library  Function Level: Coded basic functions/ algorithms

15. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 3. Increasing M&S-Credibility M&S- Credibility Perceived suitability Perceived correctness Correctness (specification  Implementation) ConsistencyCompleteness Verification (  Formalisms, specifications, transformation rules) Suitability (requirements, constraints) CapabilityFidelityAccuracy Validation

16. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Influences:V&V Framework Model knowledge Risk System knowledge V&V Credibility indicators Results:

17. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Multi-phase M&S-process  intermediate products, Structured problem description Model Documen- tation Objekt I n t e r a k t i o n Systemgrenze “Umwelt- objekt“ “Hauptobjekt” mit Modellattributen Conceptual model Formal model Executable model Simulation results Project Objectives Problem definition System analysis Formalization Implementation Experimentation Interpretation Internal consistency & completeness Consistency of transformation (pairwise comparison)

18. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Our Approach: „The V&V Triangle“ V&V Phase Intermediate Results

19. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Model Development & V&V I t

20. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 V&V - Process „Tailoring“ I t

21. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Our Contribution to International Efforts on V&V:  ITOP:International Test Operations Procedure (Working Group of Experts WGE 7.2: V&V)  NMSG:Nato Modeling and Simulation Group (019 Task Group 016: “V&V of Federations”)  SISO PDG:SISO Product Development Group: (V&V of Federations)  V&V – CONVENTION (October 20-24, 2004, Munich-Germany) EU-COST 285 – Action „..M&S in innovative telecommunications...“ ( 2003 – 2007)

22. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 4. Conclusions  Strategic approach: „Divide and Conquer“ !   M&S as a Multi-Phase Engineering Process:  Efficient M&S-design by component-based modeling  Credibility assurance by risk-based V&V  Summary:  How to improve M&S-credibility and efficiency for increasingly complex simulation applications?

23. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Focus of our research efforts  Model engineering process:  Re-useability of model components  Tailored model documentation  (Risk analysis)  tailored V&V (&A)  M&S performance & reliability:  Parallel and distributed M&S  (Collaborative modeling)

24. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 I t Thank you very much for your interest and attention!

25. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 M&S – Source of Knowledge and Expertise Modeling Expertise (HW-)SW- Expertise Experimental Design and Analysis Domain Knowledge User- Knowledge System Analysis Model Formalization Implementation Experimentation Problem Definition

26. Axel Lehmann NATO Advanced Research Workshop Velingrad, Bulgaria, Oct. 24, 2006 Example: „Booking System“ Formal Model (Version 1): queuing net