1. Software Independent Verification and Validation (IV&V)
Source Copyright & Courtesy
NASA IV&V Facility Fairmont, West Virginia Judith N. Bruner Acting Director

2. Why are we discussing IV&V?

3. Setting the Stage
In the 90s, the Commanding General of theArmy’s Operational Test and Evaluation Agency noted that 90 percent of systems that were not ready for scheduled operational tests had been delayed by immature software.

4. The Standish Group examined 8,380 Software Projects.
Software “Chaos”
The Standish Group examined 8,380 Software Projects.
53% “Challenged”
16% Successful
Over budget -
by 189%
- In Budget
- Late by 222%
- On Time
- Missing 39% of
Capabilities
- Meets
Requirements
- User involved
Note: For Large Companies
31% Cancelled
- 9% were Successful
- Development
- 61.5% Challenged -
Over budget
by 178%
- Late by 230%
- Missing 58% of Capabilities
- 29.5% were Cancelled

5. Error Densities 23% 68% 9% Design & Implementation Requirements
Specification
Installation &
Commissioning
68% 9%

6. Increasing Cost of Changes
The cost to correct an software error multiplies during the
development lifecycle.
Cost scale factor
(Normalized to Requirements Phase)

7. What is IV&V?

8. Independent Verification and Validation (IV&V)
Technical: IV&V prioritizes its own efforts
Managerial: Independent reporting route to Program Management
Financial: Budget is allocated by program and controlled at high level such that IV&V effectiveness is not compromised
Verification (Are we building the product right?)
The process of determining whether or not the products of a given phase of the software development cycle fulfill the requirements established during the previous phase
Is internally complete, consistent and correct enough to support next phase
Validation (Are we building the right product?)
The process of evaluating software throughout its development process to ensure compliance with software requirements. This process ensures:
Expected behavior when subjected to anticipated events
No unexpected behavior when subjected to unanticipated events
System performs to the customer’s expectations under all operational conditions

9. Independent Verification & Validation
Software IV&V is a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle
Adapted to characteristics of the target program

10. How is IV&V done?

11. IV&V Activities Throughout Lifecycle
Requirements Phase
System Reqts Analysis
S/W Reqts Analysis
Interface Analysis
Process Analysis
Technical Reviews & Audits
Design Phase
Design Analysis
Test Program Analysis
Supportability Analysis
Code Phase
Code Analysis
Test Phase
Independent Test
Verify
Validate
Phase-dependent tasks
Planning analyses
Software development, verification, reconfiguration plans
Requirements analyses
Command & data flow
Interface
Horizontal architectural requirements
Phasing (staging/reconfiguration)
Requirements validation
Design analyses
Critical algorithms
Timing & sizing
Code analyses
Test analyses
Plan, procedure, & report
Unit, integration, interface, acceptance and system
Phase-independent tasks
Catastrophic/mission critical/high risk functions identification
Traceability matrix
Issue/Discrepancy tracking and reporting
Analytic studies
Loading
Metrics
Schedule

12. IV&V Life Cycle Functions
IV&V Process provides tools and analysis procedures appropriate to each phase of the software development life cycle:
Formulation Phase:
Is development process sound, repeatable, and managed?
Requirements Phase:
Verify that system and software requirements are correct, complete, traceable and testable
Analyze system-level requirements: Are test plans and acceptance criteria sufficient to validate system requirements and operational needs?
Are testing methods sufficient to verify and validate software requirements?
Are the correct software development, management, and support processes in place?
Design Phase:
Does the design support the requirements?
Are test plans and test environments sufficient to verify and validate software and operational requirements?
Does the design have any characteristics that will cause it to fail under operational scenarios? What solutions are appropriate?

13. IV&V Life Cycle Functions (cont.)
Typical IV&V functions by Software life-cycle phase (cont.):
Coding Phase:
Does the code reflect the design?
Is the code correct?
Verify that test cases trace to and cover software requirements and operational needs
Verify that software test cases, expected results, and evaluation criteria fully meet testing objectives
Analyze selected code unit test plans and results to verify full coverage of logic paths, range of input conditions, error handling, etc.
Test Phase:
Analyze correct dispositioning of software test anomalies
Validate software test results versus acceptance criteria
Verify tracing and successful completion of all software test objectives
Operational Phase:
Verify that regression tests are sufficient to identify adverse impacts of changes

14. IV&V Testing Involvement
IV&V identifies deficiencies in program’s test planning
Program changes their procedures to address deficiencies vice IV&V independently test
IV&V may independently test highly critical software using an IV&V testbed
Whitebox
Stress
Endurance
Limit
Developer motivated to show software works
IV&V attempts to break software

15. IV&V Process

16. CARA Scoring Methodology
Criticality:
Catastrophic=4
Critical=3
Moderate=2
Low=1
Performance and Operations
Safety
Cost/schedule
Category
Rating
Average Criticality
Risk:
High=3
Complexity
Technology Maturity
Reqts Dfn & Stability
Testability
Developer Experience
Average Risk
CARA score
For each Software Function:
Set IV&V Analysis Level (IAL) Thresholds
IAL CARA Score
None: 1 < CARA < 2
Limited (L): 2 < CARA < 5
Focused (F): 5 < CARA < 8
Comprehensive (C): 8 < CARA < 12

17. Sample Criticality Evaluation Criteria
CARA Criticality

18. Sample Risk Driver Criteria
CARA Risk

19. Why perform IV&V?

20. IV&V Benefits Technical Management Better Visibility into
Better software/system
Performance
Higher Confidence in
Software Reliability
Compliance between
Specs & Code
Criteria for Program
Acceptance
Better Visibility into
Development
Better Decision Criteria
Second Source Technical
Alternative
Reduced maintenance cost
Reduced Frequency of
Operational Change

21. IV&V Key Points IV&V works with the Project
Goal is project success
IV&V is an engineering discipline
IV&V processes are defined and tailored to the specific program
Mission, operations and systems knowledge is used to perform engineering analyses of system components
IV&V is most effective when started early
70% of errors found in testing are traceable to problems in the requirements and design
IV&V works problems at the lowest possible level
Primarily work via established informal interfaces with the development organization - working groups, IPTs, etc.
Elevate issues only when necessary

22. IV&V Approach Efficiently Mitigates Risk
It is not necessary or feasible to perform all IV&V analyses on all software functions
IV&V resources allocated to reduce overall exposure to operational, development, and cost/schedule risks
Software functions with higher cirticality and development risk receive enhanced levels of analysis (‘CARA’ process)
Systems analyses performed to reduce costly interface and integration problems
Process analyses performed to verify ability to produce desired result relative to program plans, needs and goals
IV&V working interfaces promote timely problem resolution
Proactive participation on pertinent development teams
Emphasis on early identification of technical problems
Engineering recommendations provided to expedite solution development and implementation

23. Analyses Are Value Added and Complementary - Not Duplicative
Analyses performed from a systems perspective considering mission needs and system use, hazards and interfaces
Discipline experts assigned to perform analysis across all life cycle phases
Horizontal specialty skills are matrixed across IV&V functional teams to verify correct systems integration
Specialized tools and simulations perform complex analyses
IV&V testing activities complement developer testing enhancing overall software confidence
Developer testing focuses on demonstrating nominal behavior, IV&V testing activities try to break the software
Overall program integration, test and verification approach analyzed for completeness, integrity and effectiveness

24. Thanks…