P1516.4: VV&A Overlay to the FEDEP 20 September 2007 Briefing for the VV&A Summit Simone Youngblood Simone Youngblood M&S CO VV&A Proponency Leader

Current VV&A Standards Initiatives VV&A FEDEP Overlay VV&A Templates

VV&A FEDEP Overlay

Background VV&A Product Development Group (PDG) initiated in Spring 2004 VV&A Overlay Drafting Group had first meeting in Winter 2004 Draft overlay standard development progress briefed in Spring 2004, Fall 2004, Spring 2005, Summer 2005 and Fall 2005 Draft overlay standard released to the VV&A PDG for comment on 31 March - 15 May 2006 Drafting Group met to develop comment resolution recommendations on 5-9 June 2006 VV&A PDG met to approve comment resolutions on 28 June 2006 Revised draft standard released to the VV&A PDG for comment on 4 August - 4 September 2006 Drafting Group met to develop comment resolution recommendations on 5-9 September 2006 VV&A PDG meets to decide recommended comment resolutions on 14 September 2006

Scope Identifies and describes the recommended verification, validation and accreditation (VV&A) processes that one should perform to assure the acceptability and utility of federations for particular purposes. Identifies and describes the information feeding and resulting from those processes as well as the relationships between the High Level Architecture (HLA) Federation Development and Execution Process (FEDEP) and the VV&A processes and their respective information products. Defines those terms uniquely needed to characterize the FEDEP VV&A overlay (e.g., federation conceptual model, interoperability, representational requirements). Takes special care to use and build upon existing standards, standard terms and their definitions whenever possible. Focuses upon the VV&A processes that apply to federations. Does not address the VV&A processes associated with individual simulations (federates) but does consider using the information produced by those processes. Does not describe individual techniques that one might employ to execute the VV&A processes for federations.

Draft VV&A Overlay Standard Outline Introduction Scope Purpose References Definitions, Abbreviations & Acronyms –Definitions –Abbreviations & Acronyms Roles & Responsibilities –Introduction to VV&A Roles & Responsibilities –Role Mapping to Overlay Activities Assumptions & Tailoring –Assumptions –Tailoring Overlay: Top Level View Overlay: Detailed View –Phase 1 - Verify Federation Objectives –Phase 2 - V&V Federation Conceptual Model –Phase 3 - Verify Federation Design –Phase 4 - Verify Federation Development Products –Phase 5 - Validate & Accept Federation –Phase 6 - V&V Federation Outputs –Phase 7 - Consolidate Federation VV&A Products Conclusion

Some Terms Important to the VV&A Overlay The drafting committee encountered several terms while drafting the VV&A Overlay to the FEDEP that were relevant to federation VV&A: The drafting committee took special care to use the definitions of terms in the FEDEP and the well respected definitions from other sources whenever possible. They only judiciously crafted new definitions when no existing definition captured the meaning of the terms as they were used in the overlay (marked by an *). interoperability parameter referent representational requirements results sampling strategy * risk scenario substantive interoperability uncertainty * traceability validation verification acceptability criteria * acceptance * accreditation credibility * error characteristics * federate federation federation conceptual model federation objectives federation object model federation requirements * fidelity *

Overview of the Draft Overlay Activities Phase 1 - Verify Federation Objectives Activity 1.1 -Support Identifying User/Sponsor Needs Activity 1.2 -Plan Accreditation Activities Activity 1.3 -Support Developing Federation Objectives Activity 1.4 -Contribute to Verifying Federation Objectives Activity 1.5 -Assemble Federation Referent Activity 1.6 -Define Federation Acceptability Criteria Activity 1.7 -Plan V&V Activities Phase 2 - V&V Conceptual Model Activity 2.1 -Support Developing Federation Scenarios Activity 2.2 -Contribute to Verifying Federation Scenarios Activity 2.3 -Support Developing Federation Conceptual Model Activity 2.4 -Contribute to Verifying Federation Conceptual Model Activity 2.5 -Validate Federation Conceptual Model Activity 2.6 -Support Developing Federation Requirements Activity 2.7 -Contribute to Verifying Federation Requirements Phase 3 - Verify Federation Design Activity 3.1 -Support Selecting Federates Activity 3.2 -Support Preparing Federation Design Activity 3.3 -Contribute to Verifying Federation Design Activity 3.4 -Support Preparing Federation Development and Execution Plan Phase 4 - Verify Federation Development Products Activity 4.1 -Support Developing FOM Activity 4.2 -Contribute to Verifying FOM Activity 4.3 -Support Establishing Federation Agreements Activity 4.4 -Contribute to Verifying Federation Agreements Activity 4.5 -Support Implementing Federate Designs Activity 4.6 -Support Implementing Federation Infrastructure Activity 4.7 -Contribute to Verifying Federation Infrastructure Activity 4.8 -V&V Federation Data Sets Phase 5 - Validate & Accept Federation Activity 5.1 -Support Planning Federation Execution Activity 5.2 -Support Integrating Federation Activity 5.3 -Contribute to Verifying Integrated Federation Activity 5.4 -Support Testing Federation Activity 5.5 -Validate Integrated Federation Results Activity 5.6 -Perform Acceptance Assessment Phase 6 - V&V Federation Outputs Activity 6.1 -Support Executing Federation Activity 6.2 -Contribute to Verifying Raw Execution & Derived Output Activity 6.3 -Validate Federation Outputs Phase 7 - Consolidate Federation VV&A Products Activity 7.1 -Support Analyzing Data Activity 7.2 -Prepare Federation VV&A Products for Reuse

Top Level View of the VV&A Overlay to the FEDEP FEDEP VV&A Overlay to the FEDEP FEDEP INFORMATION & PRODUCTS Plan, Integrate & Test Federation Plan, Integrate & Test Federation Step 5 Validate & Accept Federation Validate & Accept Federation Phase 5 Validate the integrated federation results against the Federation Acceptability Criteria and against the Federation Referent. Formulate federation acceptance/ accreditation recommen- dations. Execute Federation & Prepare Outputs Execute Federation & Prepare Outputs Step 6 V&V Federation Output V&V Federation Output Phase 6 Verify the federation outputs internally. Validate federation output against the Federation Acceptability Criteria and against the Federation Referent. Define Federation Objectives Define Federation Objectives Verify Federation Objectives Phase 1 Verify the Federation Objectives with the User /Sponsor. Define the Federation Acceptability Criteria. Assemble the Federation Referent. Plan the VV&A activities. Step 1 Perform Conceptual Analysis Perform Conceptual Analysis V&V Federation Conceptual Model V&V Federation Conceptual Model Phase 2 Verify the Federation Conceptual Model internally. Validate the Federation Conceptual Model against the Federation Referent and against the Federation Acceptability Criteria. Step 2 Design Federation Design Federation Verify Federation Design Verify Federation Design Phase 3 Verify the Federation Design internally. Verify the Federation Design against the Federation Conceptual Model. Step 3 Develop Federation Develop Federation Verify Federation Development Products Verify Federation Development Products Phase 4 Verify the federation development products internally. Verify the federation development products against the Federation Conceptual Model. Step 4 Analyze Data & Evaluate Results Analyze Data & Evaluate Results Consolidate Federation VV&A Products Consolidate Federation VV&A Products Phase 7 Collect all VV&A products and lessons learned. Assemble them into a VV&A package to support federation reuse. Step 7

Detailed View of the VV&A Overlay to the FEDEP Federation VV&A products Define Federation Objectives Perform Conceptual Analysis Develop Federation Plan, Integrate and Test Federation Execute Federation and Prepare Outputs Analyze Data and Evaluate Results Design Federation verified Federation Objectives validated Conceptual Model verified Federation design verified Federation development products accepted Federation validated Federation output Verify Federation Objectives 1.0 V&V Federation Conceptual Model 2.0 Verify Federation Design 3.0 Validate & Accept Federation 5.0 V&V Federation Output 6.0 Consolidate Federation VV&A Products 7.0 Verify Federation Development Products 4.0 federation output V&V results federation development verification results federation validation and acceptance results federation conceptual model V&V results Federation Acceptability Criteria, Federation Referent, VV&A Plans federation design verification results

Tasks Associated with Each VV&A Overlay Phase Verify Federation Design 3.0 Support Selecting Federates Support Preparing Federation Design Contribute to Verifying Federation Design Support Preparing Federation Development & Execution Plan Verify Federation Objectives 1.0 Support Identifying User/Sponso r Needs Plan Accreditation Activities Contribute to Verifying Federation Objectives Support Developing Federation Objectives Assemble Federation Referent Define Federation Acceptability Criteria Plan V&V Activities V&V Federation Conceptual Model 2.0 Support Developing Federation Scenarios Contribute to Verifying Federation Scenarios Contribute to Verifying Federation Requirements Support Developing Federation Requirements Support Developing Federation Conceptual Model Contribute to Verifying Federation Conceptual Model Validate Federation Conceptual Model Validate & Accept Federation 5.0 Support Planning Federation Execution Support Integrating Federation Contribute to Verifying Integrated Federation Support Testing Federation Perform Acceptance Assessment Validate Integrated Federation Results V&V Federation Output 6.0 Support Executing Federation Contribute to Verifying Raw Execution and Derived Output Validate Federation Outputs Consolidate Federation VV&A Products 7.0 Support Analyzing Data Prepare Federation VV&A Products for Reuse V&V Federation Data Sets Verify Federation Development Products 4.0 Support Developing FOM Contribute to Verifying FOM Support Establishing Federation Agreements Contribute to Verifying Federation Agreements Contribute to Verifying Federation Infrastructure Support Implementing Federate Designs Support Implementing Federation Infrastructure

Phase 1 - Verify Federation Objectives Develop Objectives Identify User/Sponsor Needs verified Federation Objectives Federation Acceptability Criteria Federation Referent Federation V&V Plan Federation Accreditation Plan User/Sponsor Needs Support Identifying User/Sponsor Needs 1.1 Plan Accreditation Activities 1.2 Support Developing Federation Objectives 1.3 Contribute to Verifying Federation Objectives 1.4 Assemble Federation Referent 1.5 Define Federation Acceptability Criteria 1.6 Plan V&V Activities 1.7

Phase 2 - V&V Federation Conceptual Model Develop Federation Requirements verified Federation Scenarios verified Federation Objectives verified Federation Requirements validated Federation Conceptual Model Federation Referent Federation Acceptability Criteria Support Developing Federation Scenarios 2.1 Contribute to Verifying Federation Scenarios 2.2 Support Developing Federation Conceptual Model 2.3 Contribute to Verifying Federation Conceptual Model 2.4 Validate Federation Conceptual Model 2.5 Contribute to Verifying Federation Requirements 2.7 Support Developing Federation Requirements 2.6 Develop Scenario Develop Federation Conceptual Model

Phase 3 - Verify Federation Design Prepare Plan (FDEP) Select Federates Prepare Federation Design Federation Acceptability Criteria validated Federation Conceptual Model updated FDEP Federation Accreditation Plan Federation V&V Plan verified Federation design Support Selecting Federates 3.1 Support Preparing Federation Design 3.2 Contribute to Verifying Federation Design 3.3 Support Preparing Federation Development & Execution Plans 3.4

Phase 4 - Verify Federation Development Products Support Establishing Federation Agreements Support Developing FOM Support Implementing Federate Designs Develop FOM Establish Federation Agreements Implement Federate Designs Data Sets Support Implementing Federation Infrastructure Implement Federation Infrastructure verified federation design products validated Federation Conceptual Model existing federate data sets verified scenarios verified FOM verified Federation Agreements validated federates verified federation infrastructure supporting databases validated data sets Federation Referent Federation Acceptability Criteria Contribute to Verifying Federation Agreements 4.4 Contribute to Verifying Federation Infrastructure 4.7 Verify and Validate Data Sets 4.8 Contribute to Verifying FOM 4.2

Phase 5 - Validate and Accept Federation Plan Execution Federation Referent Federation Acceptability Criteria V&V activity results verified integrated federation acceptance recommendation validated Federation Federation validation results Integrate Federation Test Federation Support Planning Federation Execution 5.1 Support Integrating Federation 5.2 Contribute to Verifying Integrated Federation 5.3 Support Testing Federation 5.4 Validate Integrated Federation Results 5.5 Perform Acceptance Assessment 5.6

Phase 6 - V&V Federation Output Execute Federation Prepare Federation Outputs Federation Referent Federation Acceptability Criteria federation output validation results federation outputs Validate Federation Outputs 6.3 Support Executing Federation Contribute to Verifying Raw Execution and Derived Output

Phase 7 - Consolidate Federation VV&A Products Analyze Data Evaluate and Feedback Results federation outputs V&V activity products analyzed data reusable VV&A products Support Analyzing Data 7.1 Prepare Federation VV&A Products for Reuse 7.2

P Recommended Practice for VV&A of a Federation, an Overlay to the HLA Federation Development and Execution Process Initial Ballot Ballot Open Date: 04/17/2007Ballot Close Date: 05/17/2007 RESPONSE RATE This ballot has met the 75% returned ballot requirement 55 eligible people in this ballot group. 44 affirmative votes 3 negative votes with comments 0 negative votes without comments 5 abstention votes 52 votes received = 95% returned, 10% abstention APPROVAL RATE The 75% affirmation requirement is being met 44 affirmative votes 3 negative votes with comments 47 votes = 94% affirmative

P Recommended Practice for VV&A of a Federation, an Overlay to the HLA Federation Development and Execution Process Recirculation Ballot Ballot Open Date: 06/26/2007Ballot Close Date: 07/06/2007 RESPONSE RATE This ballot has met the 75% returned ballot requirement 55 eligible people in this ballot group. 45 affirmative votes 2 negative votes with comments 0 negative votes without comments 5 abstention votes 52 votes received = 95% returned, 10% abstention APPROVAL RATE The 75% affirmation requirement is being met. 45 affirmative votes 2 negative votes with comments 47 votes = 96% affirmative

Summary of Accomplishments Developed a succinct and consistent model of the federation VV&A process that guided the the overlay’s formulation Crafted definitions for several important terms (e.g., federation requirements, credibility) where no relevant definitions existed in other sources Integrated the results from the preliminary NATO VV&A Overlay with the initial draft of the SISO VV&A Overlay to the FEDEP Clearly distinguished three important classes of tasks that involve the VV&A Team: –Support - FEDEP tasks in which the VV&A Team should participate –Contribute - verification tasks that the VV&A Team should monitor and supplement where necessary –VV&A unique - tasks for which the VV&A Team are completely responsible Clarified the VV&A responsibilities for Steps 6 & 7 of the FEDEP, the life after federation acceptance Reached agreement between international representatives (Canada, France, Germany, Sweden, United Kingdom & USA) on the structure and details of the VV&A overlay to the FEDEP Ballotted P through the IEEE process

VV&A Templates

Mission Statement Enable expanded M&S reuse by building the foundation for consistent V&V information to support accreditation decisions.

Outlines of the Four Core VV&A Documents Accreditation PlanV&V PlanV&V ReportAccreditation Report Executive Summary 1 Problem Statement 2 M&S Requirements and Acceptability Criteria 3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts 4 Accreditation Methodology4 V&V Methodology4 V&V Task Analysis4 Accreditation Assessment 5 Accreditation Issues5 V&V Issues5 V&V Recommendations5 Accreditation Recommendations 6 Key Participants 7 Planned Accreditation Resources 7 Planned V&V Resources7 Actual V&V Resources7 Actual Accreditation Resources 8 V&V Lessons Learned8 Accreditation Lessons Learned Suggested Appendices: A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G Accreditation Programmatics H Distribution List Suggested Appendices: A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G V&V Programmatics H Distribution List I Accreditation Plan Suggested Appendices: A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G V&V Programmatics H Distribution List I V&V Plan J Test Information Suggested Appendices: A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G Accreditation Programmatics H Distribution List I Accreditation Plan J V&V Report

Templates Status Draft MIL Standard entitled Department of Defense Standard Practice: Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A) Documentation Templates DSP review closed 17 July 07 Comment adjudication completed 23 August 07