1. 2012 Spring Simulation Interoperability Workshop
IEEE Std Extending Verification and Validation from Software to Systems
March 28, 2012
2012 Spring Simulation Interoperability Workshop
VV&A Forum
Orlando, FL
Edward A. Addy, PhD, PMP
Northrop Grumman
Copyright © 2012 Northrop Grumman Systems Corporation.  All rights reserved.
Log #DSD

2. Acknowledgement
The work described in this presentation is being performed by the IEEE P1012 Working Group, which is chartered by the Software & Systems Engineering Standards Committee (C/S2ESC) of the IEEE Computer Society and has voluntary participation by members of the systems and software engineering communities.
Chair Roger U. Fujii, Fujii Systems
Vice-Chair Kenneth A. Costello, NASA
Secretary Edward A. Addy, Northrop Grumman
IEEE Std was approved by ballot in December 2011 and is scheduled for publication this summer.

3. Fundamental Concepts From The House at Pooh Corner, A. A. Milne
Setting: Small, one of the relatives of Rabbit, has not been seen for a long time.
"Now," said Rabbit, "this is a Search, and I've organized it----"
"Done what to it?" said Pooh.
"Organized it. Which means--well, it's what you do to a
Search, when you don't all look in the same place at once.”
Look using a planned approach
Look in the most important places
Look using different techniques

4. Definitions of Verification and Validation
ISO/IEC/IEEE 24765:2010(E) Systems and software engineering — Vocabulary
Verification – The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase
Validation – The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements
Build the system correctly
Build the correct system

5. Definitions of Verification, Validation and Accreditation (M&S Perspective)
U.S. DoD Instruction
Verification – The process of determining that a model or simulation implementation and its associated data accurately represent the developer’s conceptual description and specifications
Validation – The process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model
Accreditation – The official certification that a model or simulation and its associated data are acceptable for use for a specific purpose
Build the system correctly
Build the correct system
Should the system be used?

6. 1012-2004 Software Verification and Validation (Current version of the Standard)
Software V&V tasks, along with the activities for each task, are identified for software life cycle processes
maps V&V tasks to the six primary life cycle processes of ISO/IEC 12207:1995 Standard for Information Technology Software Life Cycle Processes
Management Process − Development Process
Acquisition Process − Operation Process
Supply Process − Maintenance Process
Minimum tasks are mapped to Software Integrity Levels based on Criticality Analysis
also provides
Overview of the Standard, Definitions, General Description of V&V Processes
Reporting and Administrative Requirements, Software V&V Plan Outline
Annexes: A Risk-Based Integrity Level Scheme; Definition of Independent V&V; V&V of Reuse Software; V&V Measures; Example of V&V Relationships to Other Project Responsibilities; Optional V&V Tasks

7. ISO/IEC 15288:2008 Systems and Software Engineering – System Life Cycle Processes
Agreement
Project
Technical
Acquisition
Project Planning
Stakeholder Reqmnts Def
Requirements Analysis
Supply
Project Assessment and Control
Arch Design
Organizational
Project-Enabling
Implementation
Decision Mgt
Integration
Risk Mgt
Life Cycle Model Mgt
Verification
Configuration Mgt
Infrastructure Mgt
Transition
Information Mgt
Project Portfolio Mgt
Validation
Measurement
Human Resource Mgt
Operation
Maintenance
Quality Mgt
Disposal

8. SW Qualification Testing
ISO/IEC 12207:2008 Systems and Software Engineering – Software Life Cycle Processes
System Context
SW Implementation
SW Support
Processes from ISO/IEC that provide a system context for dealing with a standalone software product or service or a software system
SW Implementation
SW Doc Mgt
SW Reqmnts Analysis
SW Config Mgt
SW Arch Design
SW Qual Assurance
SW Detailed Design
SW Verification
SW Construction
SW Validation
SW Integration
SW Review
SW Qualification Testing
SW Audit
SW Problem Resolution
SW Reuse
Reuse Asset Mgt
Reuse Program Mgt
Domain Engineering

9. Look using a planned approach
Verification and Validation Processes in ISO/IEC and ISO/IEC 12207
ISO/IEC 15288: 2008 provides about two pages each on the Verification Process and the Validation Process – Tasks to Plan, Perform
ISO/IEC 12207: 2008 provides about three pages on the Verification Process and about two on the Validation Process
Determine if Verification or Validation is necessary, and if so, determine the scope and develop a plan
Verify requirements, design, code, integration and documentation
Validate by testing (although “Other means besides testing (such as analysis, modeling, simulation, etc.) may be employed for validation”)
The V&V Standard ( ) provides a conforming instance of the System Verification and System Validation Processes of ISO/IEC 15288: 2008 and of the Software Verification and Software Validation Processes of ISO/IEC 12207: 2008
IEEE 1012 provides much more detail, particularly useful for systems requiring a high level of assurance
Look using a planned approach

10. 1012-2012 System and Software Verification and Validation (Revised Standard)
Includes V&V tasks and activities for systems and hardware in addition to software
Maps to current ISO/IEC Standards
System V&V tasks  Technical Life Cycle Processes of ISO/IEC 15288:2008
Software V&V tasks  Software Implementation Processes of ISO/IEC 12207:2008
Common V&V tasks  Agreement Life Cycle Processes and to the Project Planning and Configuration Management Processes of ISO/IEC 15288:2008
Continues the approach of Criticality Analysis to determine Integrity Levels
Continues to provide
Overview of the Standard, Definitions, General Description of V&V Processes
Reporting and Administrative Requirements, V&V Plan Outline
Annexes from 2004 Version
Adds new Annexes
Environmental Factors; V&V of System, Software and Hardware Integration; Hazard, Security and Risk Analysis; Example of Assigning and Changing the System Integrity Level of “Support System Functions”

11. 1012-2012 Addresses System, Software and Hardware V&V
Overview of the Standard, Definitions, General Description of V&V Processes and Relationships to other Processes
Common Verification and Validation
Description of V&V Requirements Common to System, Software and Hardware
System Verification and Validation
Description of the System-level V&V Requirements, primarily against the ISO/IEC Technical Processes
Software Verification and Validation
Description of the Software V&V Requirements, primarily against the ISO/IEC Software Implementation Processes
Hardware Verification and Validation
Description of the Hardware V&V Requirements, generally based on processes described in the INCOSE Systems Engineering Handbook
Tables
Table 1 – V&V Tasks
Sectioned into Common, System, Software and Hardware
Each section separated into V&V activities based on life cycle processes
Table 2 – Minimum V&V Tasks for each Integrity Level
Table 3 – Optional V&V Tasks
Reporting and Administrative Requirements, V&V Plan Outline
Annexes

12. System Development Occurs using a Recursive Application of Life Cycle Processes
The System Implementation Process results in a realized system element, i.e., a lower-level system or a software or hardware element.
Life cycles are recursive and iterative.
V&V addresses the collective results of:
Environmental Requirements
Operational Issues
Legal and Policy Constraints
Security and Safety
Training and Human Interface
System-of-Interest
Implementation Process
System V&V
System
System
Software element
Implementation Process
Implementation Process
Software V&V
System V&V
System V&V
Software element
Hardware element
Software element
Hardware element
Hardware element
Software V&V
Hardware V&V
Software V&V
Hardware V&V
Hardware V&V

13. System Life Cycle with Embedded Software and Hardware Life Cycles

14. Use of IEEE Std
Revised standard is designed so that any combination of systems, software and/or hardware V&V can be performed in conformance with the standard.
Conduct System V&V by using
Common V&V requirements
Common V&V
System V&V requirements
Portions of Software or Hardware V&V
for high Integrity Levels if V&V is
not being performed on those
components
System V&V
Conduct Software V&V by using
Common V&V requirements
Software V&V requirements
Software V&V
Conduct Hardware V&V by using
Common V&V requirements
Hardware V&V requirements
Hardware V&V

15. Criticality Analysis Determines Integrity Levels that are Mapped to V&V Tasks
Conformance with requires Criticality Analysis to determine the Integrity Level of the system and each element
Integrity Level is based on the importance of the system to the user and acquirer
Criticality Analysis considers various aspects such as complexity, risk, safety level, security level, desired performance, reliability, or other project-unique characteristics
uses Integrity Levels to determine the minimum V&V tasks to be performed
V&V tasks are assigned based on Integrity Level, with more tasks and activities being conducted at higher Integrity levels
Integrity Levels are also used to determine the rigor and level of intensity for V&V tasks and activities
The assigned Integrity Levels may change as the system evolves, so Criticality Analysis is repeated during development
Look in the most important places, and make sure you know when there is a change in what is important.

16. Example Software V&V Tasks & Activities
From Table 1c— V&V tasks, inputs, and outputs
9.1 Activity: Concept V&V (Software, Process: Software Requirements Analysis)
V&V Tasks
Required Inputs
Required Outputs
Concept Documentation Evaluation
V&V subtasks
Concept documentation
System architectural design
Supplier development plans and schedules
User needs
Acquisition needs
Task report(s) – Concept documentation evaluation
Anomaly report(s)
(4) Criticality Analysis
Concept documentation (system requirements)
Developer integrity level assignments
Task report(s) – Criticality analysis
(5) Hazard Analysis
Task report(s) – Hazard analysis

17. Example Minimum V&V Tasks Assigned to Integrity Levels
From Table 2c— Minimum V&V tasks assigned to each integrity level for software V&V
V&V Activities
Activity: Software Concept V&V
(see 9.1)
Activity: Software Requirements V&V (see 9.2)
Integrity Levels
Levels 1 2 3 4
Concept Documentation Evaluation X
Criticality Analysis
Design Evaluation
Evaluation of New Constraints
Hardware/Software/User Requirements Allocation Analysis
Hazard Analysis
Look using different techniques

18. System V&V Task Overview (similar pattern for Software and Hardware V&V Tasks)
System Life Cycle Processes
System V&V Tasks (Minimum Requirements)
Product Evaluation & Assessment
Conformance & Scoping
Test
System & Environment Issues
Stakeholder Requirements Definition
Stakeholder Requirements Evaluation
Traceability & Criticality Analyses
Hazard, Security & Risk Analyses
Requirements Analysis
Requirements Evaluation
Interface Analysis
Architectural Design
Design Evaluation
Implementation
Implementation Strategy Assessment
System Element Implementation
Analysis
Criticality Analysis
Integration
System Integration Strategy
Assessment
Transition
Transition Strategy Evaluation
Transition Demonstration Assessment
Operations
Operating Procedure Evaluation
Maintenance
System Maintenance Strategy
System Maintenance Execution
Disposal
Disposal Plan Evaluation
Test Planning an d Execution

19. System, Software and Hardware Testing
V&V Testing by Integrity Level 1 2 3 4
V&V Software Component Testing
No action
Review
Perform
V&V Software Integration Testing
V&V Software Qualification Testing
V&V Software Acceptance Testing
Hardware
V&V Testing by Integrity Level 1 2 3 4
V&V Software Component Testing
No action
Review
V&V Software Integration Testing
V&V Software Qualification Testing
Perform
V&V Software Acceptance Testing
System
V&V Testing by Integrity Level 1 2 3 4
V&V Software Integration Testing
No action
Review
Perform
V&V Software Qualification Testing
V&V Software Acceptance Testing 19
2012 Spring SIW

20. IEEE Std ?