1. WE 0201 – PKIT2Date : November 2004Slide : 1 / 74 WEAG – THALES – JP 11.20 REVVAREVVA

2. WE 0201 – PKIT2Date : November 2004Slide : 2 / 74 Presentation Outline Project facts ; the consortium Background and rationale The REVVA concepts The REVVA generic process The REVVA products Techniques and support REVVA technical achievements REVVA future

3. WE 0201 – PKIT2Date : November 2004Slide : 3 / 74 Project Facts Purpose: The THALES JP 11.20 is a Western European Union / WEAG research program funded by 5 nations: France (lead nation), Denmark, Italy, the Netherlands, Sweden. Objectives: To develop a basis for a common methodological framework for the Verification, Validation and Accreditation (VV&A) of data, models and simulations. To be a starting point for a draft standard defining VV&A taxonomy, process and methodology. Project timeline : March 2003 – September 2004 Common Validation, Verification and Accreditation Framework for Simulation (REVVA)

4. WE 0201 – PKIT2Date : November 2004Slide : 4 / 74 The consortium The THALES JP 11.20 is conducted by a multi- national consortium led by ONERA (France). UNI-C (DK) TNO (NL) FOI – FMV (SW) DATAMAT (IT) ONERA – ALEKTO (FR)

5. WE 0201 – PKIT2Date : November 2004Slide : 5 / 74 REVVA background and rationale

6. WE 0201 – PKIT2Date : November 2004Slide : 6 / 74 REVVA Background (1) JP 11.20 REVVA ITOP AFDRG DMSO Capitalization Exchange SISO NATO RTP 11.13 Exchange Capitalization Information

7. WE 0201 – PKIT2Date : November 2004Slide : 7 / 74 REVVA Background (2) VV&A state-of-the-art: –DMSO, SISO, NATO, ITOP. M&S scientific contributions: –Systems Theory (B. Zeigler) Contributions on the concept of validation coming from the epistemology of simulation

8. WE 0201 – PKIT2Date : November 2004Slide : 8 / 74 VV&A rationale (1) Doing (only) what makes sense for customers and users in their search for trust. Trust: –identification of trust components, –support (based on project knowledge, consolidation by shared domain knowledge)

9. WE 0201 – PKIT2Date : November 2004Slide : 9 / 74 VV&A rationale (2) Trust M&S supplier skills and reputation Audit trail of experiments and uses Formal analysis of products, processes and organisation VV&A Informal Formal

10. WE 0201 – PKIT2Date : November 2004Slide : 10 / 74 VV&A rationale (3) ProcessesOrganisationProducts Needs Purpose Formal Analysis Acceptance Validation Verification

11. WE 0201 – PKIT2Date : November 2004Slide : 11 / 74 VV&A rationale (4) Activities driven and prioritised by: Requirements (acceptance requirements) Identification / perception of risks Organisational needs, risks and benefits Costs

12. WE 0201 – PKIT2Date : November 2004Slide : 12 / 74 REVVA Concepts

13. WE 0201 – PKIT2Date : November 2004Slide : 13 / 74 REVVA Terminology (1) Focus on Acceptance and Validation Decomposition of VV&A in Verification, Validation and Acceptance; (temporal) independence of each concept. –Verification: Model Verification. –Validation: Model Validation within an Experimental Frame (EF) –Acceptance: Before use / After use

14. WE 0201 – PKIT2Date : November 2004Slide : 14 / 74 REVVA Terminology (2) Correctness: –The property of a simulation model to comply with formal rules and bodies of reference information for its representation and for the transformation of its representation into another one. Verification: –The process which is used to construct, under a set of time, cost, skills, and organisational constraints a justified belief about model correctness.

15. WE 0201 – PKIT2Date : November 2004Slide : 15 / 74 REVVA Terminology (3) Validity: –The property of a simulation model to have, within a specific experimental frame, a behaviour which is indistinguishable under a set of validation criteria from the behaviour of the System of Interest. Validation: –The process which is used to construct, under a set of time, cost, skills, and organisational constraints a justified belief about model validity.

16. WE 0201 – PKIT2Date : November 2004Slide : 16 / 74 REVVA Terminology (4) Acceptance: –The informed decision of a M&S customer to use the results of a simulation for a specific purpose. –The informed decision of a M&S customer to use a simulation, within an experimental frame, for a specific purpose

17. WE 0201 – PKIT2Date : November 2004Slide : 17 / 74 REVVA Assumptions Acceptance (and Validation) should: –be purpose oriented, –be development neutral, –be technology neutral, –recognise the impact of organisations, –recognise the impact of timeframe of M&S and VV&A, –rely on data and knowledge management.

18. WE 0201 – PKIT2Date : November 2004Slide : 18 / 74 A comprehensive view The "Real World" Objectives Needs Decisions Actions Problem Formalisation Problem solution M&S Questions M&S Answers System Requirements M&S Enabling Products The "Problem World" The "M&S World" The "M&S Product World"

19. WE 0201 – PKIT2Date : November 2004Slide : 19 / 74 Methodological hints Independence Incremental nature of activities and results Focus on the intended purpose Objective assessment (Acceptance Criteria) Careful construction of a justified belief Tailoring

20. WE 0201 – PKIT2Date : November 2004Slide : 20 / 74 Tailoring The EF used for Acceptance (and for Validation) is derived from the Acceptance Requirements All the activities are planed, designed and controlled by them. The Acceptance Requirements are not (necessarily) the M&S Requirements ; they are specific of each use and purpose EF M Question Answer

21. WE 0201 – PKIT2Date : November 2004Slide : 21 / 74 Impact of time t ||| Before M&S use1 Before M&S use2 After M&S use1 t1t2t3 Validation t1Validation t2Validation t3 Verification is (mostly/hopefully) a one-shot process Validation is a permanent/recurrent issue Acceptance and Validation are purpose oriented and tailored

22. WE 0201 – PKIT2Date : November 2004Slide : 22 / 74 The REVVA Generic Process

23. WE 0201 – PKIT2Date : November 2004Slide : 23 / 74 Acquire Information Develop ToVV Conduct V&V Assess Evidence Assess Evidence Integration Evaluate V&V Report Develop ToA ToVV V&V Items Assessed items of evidence V&V report Acceptance Recommendation Model information and system knowledge Phase Product Intended purpose Product flow Back step The REVVA Generic Process ToA: Target of Acceptance ToVV: Target of Verification and Validation

24. WE 0201 – PKIT2Date : November 2004Slide : 24 / 74 Phase Overlaps Develop ToA Acquire Information Develop ToVV Conduct V&V Assess Items of Evidence Assess Evidence Integration Evaluate V&V Report VV&A Endeavour Duration Main effort

25. WE 0201 – PKIT2Date : November 2004Slide : 25 / 74 Develop ToA Performed by the Acceptance Agent in close cooperation with the user Develop Acceptability Criteria (AC), including Validity Criteria (VC) Document rationale for development

26. WE 0201 – PKIT2Date : November 2004Slide : 26 / 74 Intended Purpose Objectives and Sub-Objectives (Questions) (Fictive) Experimental Frames Precision Requirements for the Behavior Characteristics Develop ToA (2)

27. WE 0201 – PKIT2Date : November 2004Slide : 27 / 74 Intended Purpose AC1 AC2 Acceptability Criterion m Sub-Objective 1Sub-Objective 2Sub-Objective n AC3 ToA SO1.1

28. WE 0201 – PKIT2Date : November 2004Slide : 28 / 74 Acquire Information Led by the V&V Agent, performed by the V&V Team Get overview over the data, information, and knowledge (likely to become) available –about the Simulation Executable Model –about the System of Interest Create data, information, and knowledge repository Identify additional sources

29. WE 0201 – PKIT2Date : November 2004Slide : 29 / 74 Develop ToVV Performed by the V&V Agent, supported by the V&V Team Develop an approach to demonstrate that the Validation Criteria are passed or failed, with the data, information, and knowledge available Identify data, information, and knowledge desirable for more efficient V&V

30. WE 0201 – PKIT2Date : November 2004Slide : 30 / 74 Intended Purpose AC1 AC2 Acceptability Criterion m Sub-Objective 1Sub-Objective 2Sub-Objective n AC3 ToA ToVV SO1.1 Task Sub-Criterion m.1 SC m.1.1SC m.1.2SC m.1.3

31. WE 0201 – PKIT2Date : November 2004Slide : 31 / 74 Conduct V&V Led by the V&V Agent, performed by the V&V Team Choose techniques and tools Run the technique, e.g., –Use 2 -test to measure deviation –Use SME to assess structural decomposition Record technique, referrenced information, and result as V&V item Modify ToVV, if creation of V&V item fails

32. WE 0201 – PKIT2Date : November 2004Slide : 32 / 74 Assess Evidence Performed by the V&V Agent Review documented V&V items Assess the possibility of an erroneous result Accept V&V item as assessed item of evidence, or reject it Direct feedback loop

33. WE 0201 – PKIT2Date : November 2004Slide : 33 / 74 Assess Evidence Integration Performed by the Acceptance Agent Assemble items of evidence according to most recent ToVV Derive which VC are passed or failed –Intuitively, subjectivly –Based on methods for dealing with uncertainty Assess possibility of misperception

34. WE 0201 – PKIT2Date : November 2004Slide : 34 / 74 Intended Purpose AC1 AC2 Acceptability Criterion m Sub-Objective 1Sub-Objective 2Sub-Objective n AC3 ToA ToVV Item of Evidence IoE SO1.1 Task Sub-Criterion m.1 SC m.1.1SC m.1.2SC m.1.3

35. WE 0201 – PKIT2Date : November 2004Slide : 35 / 74 Evaluate V&V Report Performed by the Acceptance Agent Review expressiveness and completeness of AC with newly gained knowledge Weigh impact of misperception according to possibility of misperception Make acceptance recommendation

36. WE 0201 – PKIT2Date : November 2004Slide : 36 / 74 The REVVA Products

37. WE 0201 – PKIT2Date : November 2004Slide : 37 / 74 Acquire Information Develop ToVV Conduct V&V Assess Evidence Assess Evidence Integration Evaluate V&V Report Develop ToA ToVV V&V Items Assessed items of evidence V&V report Acceptance Recommendation Model information and system knowledge Phase Product Intended purpose Product flow Back step The REVVA Generic Process

38. WE 0201 – PKIT2Date : November 2004Slide : 38 / 74 Target of Acceptance (ToA) Precise specification of the Acceptability Criteria (coming from validity related functional requirements) Stakeholder oriented What exactly needs to be assessed? Important elements in the ToA nodes: requirement, MoE, constraint, level of impact, decomposition argument Acquire Information Develop ToVV Develop ToA Intended purpose

39. WE 0201 – PKIT2Date : November 2004Slide : 39 / 74 ToA (2) Building –Getting the requirements –Deriving MoE and constraints –Determining the level of impact –Validation of the ToA –Reuse Using –Part of contract –Basis for assessment report Maintaining –Due to changes in science, stakeholder changing mind, corrections in ToA found during V&V work, V&V findings may indicate more limited validity range

40. WE 0201 – PKIT2Date : November 2004Slide : 40 / 74 Model information and system knowledge Information used to determine the V&V strategy for demonstrating the models correctness and validity Identifies all sources of information and knowledge Acquire Information Develop ToVV Conduct V&V Assess Evidence Integration Model information and system knowledge Important elements: –design documents, code, modellers perception of the real world, behaviour data (of model and sub- models), conceptual model. –Real world data (referent): data on (related or similar) systems, expert opinions

41. WE 0201 – PKIT2Date : November 2004Slide : 41 / 74 Target of Verification and Validation (ToVV) V&V items: evidence required to substantiate the ToA Acceptability Criteria When necessary: further decomposition of criteria (extension of ToA) Develop ToVV Conduct V&V Assess Evidence Integration ToVV Important additional elements in the ToVV: method to use for collecting evidence, required rigor, cost estimate, aggregation instructions Tasks

42. WE 0201 – PKIT2Date : November 2004Slide : 42 / 74 V&V Items A V&V item consists of: –relevant information, –the evaluation objective, –reference information, –an evaluation technique, –evaluation result. Must be reproducible Conduct V&V Assess Evidence V&V Items

43. WE 0201 – PKIT2Date : November 2004Slide : 43 / 74 Assessed items of evidence Assessed items of evidence: –V&V agent assessment of the V&V result, –the acceptance or rejection of each V&V item, –probative force (depending on method or technique and reference information) Assess Evidence Assess Evidence Integration Assessed items of evidence

44. WE 0201 – PKIT2Date : November 2004Slide : 44 / 74 V&V report The assessed items of evidence assembled and integrated based on the ToVV Links the assessed items of evidence to the Acceptability Criteria Indicates if –evidence is considered sufficient to pass an acceptability criterion –V&V items for a criterion are missing or too weak, –disproving evidence was found Assess Evidence Integration Evaluate V&V Report V&V report

45. WE 0201 – PKIT2Date : November 2004Slide : 45 / 74 Acceptance Recommendation The final recommendation on acceptance Based on the ToA Residual uncertainty after V&V –from probative forces, –use level of impact to determine recommendation The recommendation states (with consequences for M&S products utility): –if required evidence is missing or too weak, –if a weakness in the ToA was detected Evaluate V&V Report Acceptance Recommendation

46. WE 0201 – PKIT2Date : November 2004Slide : 46 / 74 A VV&A project management view The "3 pillars" concept Process OrganisationProducts Purpose Objectives

47. WE 0201 – PKIT2Date : November 2004Slide : 47 / 74 The M&S and VV&A roles M&S stakeholders M&S suppliers VV&A agents Experts

48. WE 0201 – PKIT2Date : November 2004Slide : 48 / 74 Levels of independence DV&V: –dependent V&V; V&V performed by suppliers and accepted as it is IA: –independent assessment; V&V performed by supplier and assessed by the V&V agent IV&V: –independent V&V; V&V performed under the control of the V&V agent

49. WE 0201 – PKIT2Date : November 2004Slide : 49 / 74 Feasibility assumptions High-level organisational decision Standards of V&V documentation Incremental accumulation of knowledge: –project vs corporate knowledge management Process for knowledge revision

50. WE 0201 – PKIT2Date : November 2004Slide : 50 / 74 The REVVA Techniques and support

51. WE 0201 – PKIT2Date : November 2004Slide : 51 / 74 REVVA point of view Identification of the most important techniques and support Techniques to support decision making, … Next direction: develop techniques to support the whole process

52. WE 0201 – PKIT2Date : November 2004Slide : 52 / 74 Most important techniques and support Most importance of Requirements Engineering –Develop ToA & ToVV Design of Experiments (DOE) –Conduct V&V: calibration, sensitivity analysis & validation Computation of probative force and convincing force –Assess evidence –Assess argument Uncertainty analysis –Assess evidence –Evaluate V&V Report Decision tools –Evaluate V&V Report

53. WE 0201 – PKIT2Date : November 2004Slide : 53 / 74 Claim-Argument-Evidence Types of argument Evidence Claim Argument Deterministic – relying upon axioms, logic and proof Probabilistic – relying upon probabilities, statistical analysis Qualitative – relying upon adherence to standards, design codes

54. WE 0201 – PKIT2Date : November 2004Slide : 54 / 74 Decision Tools The model is fit for purpose The model is not fit for purpose Fact Perception ValidNot valid AcceptedOKType II error RejectedType I errorOK

55. WE 0201 – PKIT2Date : November 2004Slide : 55 / 74 Uncertainty Analysis Integrated approach for uncertainty estimation based on –Behaviour Characteristics –Propositional Logic –Hierarchical Breakdown Structures –Possibility Theory –Basic Model of Decision Theory Increases quantitative share in decision making Leaves subjectivity gaps with –Probative Forces –Impressions of confirmation and denial

56. WE 0201 – PKIT2Date : November 2004Slide : 56 / 74 p1p1 p 1 : p 1.1 p 1.2 p 1.3 + (--) p 1.1 p 1.2 p 1.3 Composition and Inference + : inference relation (p 1 p 1 : p 1 is a sufficient condition, but not a necessary one) : composition relation (p 1 is a conjunction of p 1.1 through p 1.3 ) (--) : quantification of denial impression (from (-) to (----)) (if p 1 is not verified, how strongly would it mean that p 1 is not verified?)

57. WE 0201 – PKIT2Date : November 2004Slide : 57 / 74 Completeness and consistency checking The dependencies and consistency between the sponsor needs and requirements, any structured problem description, the model (conceptual, formal or executable), the ToA and ToVV and the final simulation results are checked Contents (evidences, specifications, etc.) without requirements and requirements without contents need to be resolved

58. WE 0201 – PKIT2Date : November 2004Slide : 58 / 74 REVVA technical achievements

59. WE 0201 – PKIT2Date : November 2004Slide : 59 / 74 REVVA Project Breakdown The project includes six technical Work Packages: WP 1 : Problem Analysis WP 2 : Definition of AccreditationTarget WP 3 : VV&A Techniques WP 4 : Optimisation of Testing WP 5 : Outline for a VV&A Methodological Framework WP 6 : Evaluation of Results

60. WE 0201 – PKIT2Date : November 2004Slide : 60 / 74 Technical achievements (1) Taxonomy: Initial contribution dealing with V&V state of the art VV&A problem statement Review of acceptance criteria and levels of confidence Process models

61. WE 0201 – PKIT2Date : November 2004Slide : 61 / 74 Technical achievements (2) Emphasis on Acceptance Agent role: Target of Acceptance (ToA) and assessment of ToA satisfaction Integration of qualitative and quantitative confidence in a logical framework of argumentation Guidance for ToA and ToVV definition

62. WE 0201 – PKIT2Date : November 2004Slide : 62 / 74 Technical achievements (3) Review of existing techniques and tools resources V&V techniques state-of-the-art User guidance

63. WE 0201 – PKIT2Date : November 2004Slide : 63 / 74 Technical achievements (4) Capitalisation of data and knowledge Experience optimisation Relation with field trial data Identification of data/knowledge management issues

64. WE 0201 – PKIT2Date : November 2004Slide : 64 / 74 Deep understanding of the terminology Definition of the VV&A scope Identification of VV&A organisations depending on level of impact Definition of VV&A products and processes Technical achievements (5)

65. WE 0201 – PKIT2Date : November 2004Slide : 65 / 74 REVVA Outcomes Generic VV&A process, described by: –Methodology Reference manual (METHGU) –Users Manual (PROSPEC) Illustration by case studies Draft contributions to guidance REVVA Website (www.JP1120-REVVA.com)

66. WE 0201 – PKIT2Date : November 2004Slide : 66 / 74 Lessons learnt at methodological level Compliant with the mainstream idea that every euro spent in requirements generates high Returns on Investment (ROI) The cost driver is the criticality of the use, not the quality of the process Generic, versatile methodology: impact on the easiness of tailoring Ability to comply with formal supports for guidance or automation of the process

67. WE 0201 – PKIT2Date : November 2004Slide : 67 / 74 Lessons learnt from the case studies The REVVA application on Case Studies has shown capital points: To answer the right question: –It is important to formalise at early steps the problem to be solved by M&S use AND the question asked to VV&A. To answer the question right: –It is enabled by the derivation of the ToVV from the ToA. To construct a justified belief: –The confidence is substantiated by gathering and assessing the items of evidence.

68. WE 0201 – PKIT2Date : November 2004Slide : 68 / 74 Practical experience on case studies REVVA method helps to balance between purpose, model and data Difficult to gather the right information ToA seems to be particularly useful for large projects

69. WE 0201 – PKIT2Date : November 2004Slide : 69 / 74 REVVA communications 04S-SIW-153 : A Common Validation, Verification and Accreditation Framework for Simulations. 04S-SIW-005 : T&E Based on M&S VV&A: The Importance of Experience Capitalization. 04S-SIW-152 : Assessment Criteria and V&V Levels in JP11.20. Three papers have been presented in a special session during the Spring '04 SIW

70. WE 0201 – PKIT2Date : November 2004Slide : 70 / 74 REVVA Future

71. WE 0201 – PKIT2Date : November 2004Slide : 71 / 74 Follow-up REVVA is a first methodological step; the work will continue with a EUROPA project (REVVA2) which is to start during Spring 05.

72. WE 0201 – PKIT2Date : November 2004Slide : 72 / 74 REVVA2: Conceptual challenges Organisation and roles clarification Incremental knowledge management (i.e. repository …) Formalisation of VV&A products Formalisation of the methodology Tailoring Costs model and project management

73. WE 0201 – PKIT2Date : November 2004Slide : 73 / 74 REVVA2: Technical challenges VV&A requirements formalisation V&V evidences : collection, organisation V&V argumentation: formalisation and support V&V and decision-making under uncertainty Formal foundations for V&V tools V&V tools selection and guidance (tailoring) V&V actual processes support

74. WE 0201 – PKIT2Date : November 2004Slide : 74 / 74 REVVA2: Current status Objective: to produce draft documents for standardisation. The project timeline will be 36 months, expected start in March 2005. EUROPA MoU FR, DK, SW and NL intend to join REVVA2. The UK may consider to join after CConVV&A. The WBS are refined taking into account the roadmap defined in Munich.