1. Independent Verification and Validation (IV&V)
Independent Verification and Validation (IV&V)
"Ignorance, when voluntary, is criminal, and a man may be charged with that evil which he neglected or refused to learn how to prevent.“
Samuel Jackson

2. Content Definitions Advantages of IV&V When is IV&V Needed ?
Key Concepts
Technical, Managerial and Financial Independence
IV&V at NASA
Forms of Independence
Allocation of IV&V Effort
Selection criteria
11/16/2018

3. Background Genesis: Early Space & Missile Programs
Motivation: “No Second Chance” Performance
Example: Ariane 5
Failure Impact:
Loss of human life, impact on environment, business and pride
Example: Therac 25.
Twist: Independence of Evaluator
11/16/2018

4. Developers and IV & V Perspectives
Product
Demonstrate correct operation
Work to break it
Mindset
Optimistic
Pessimistic
Testing
Necessary evil
Inherently good
Allegiance
Boss
Customer
Source: Jose Martin, Mike Mcgee, Dean Barten, ‘Independent Verification and Validation’.
11/16/2018

5. Verification Between 2 Levels of Specifications
Software CI
Interface
System Spec
Hardware CI
11/16/2018

6. Validation Prerequisite works before IT&T Phase begins: 11/16/2018
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

7. Advantages of IV&V
Provides an objective assessment of the product during its creation,
Adds a new analytical perspective not present in the development environment,
Brings its own set of tools and techniques to bear on ensuring development accuracy and validity,
Introduces “intermediate” users of the system who serve as “beta testers” before the product goes to market,
Promotes the earlier detection of software and system errors
Significantly enhances testing and the discovery of design flaws and coding errors.
Earlier error detection translates into reduced effort and cost in removing those errors.
Reduction in development time
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

8. When is IV&V Needed ?
Real-time critical software that must work every time,
Programs having a high cost of failure
In terms of human life, national security, or money,
Software for which the cost of error detection through operational use exceeds the cost of IV&V,
Software for which the cost of maintenance and modifications exceeds the costs of IV&V.
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

9. Key Concepts
Product of each phase of development is assessed independently
i.e. not by biased developers
Assessment is not influence by management, financial or technical factors of development organization
‘Fresh viewpoint’
i.e. outside from culture of developers
Does not replace development organization QA
IV&V is the third line of defense after reviews and testing activities performed by development organization
IV&V reports to acquirer.
11/16/2018

10. Forms of Independence Classical IV&V
Embodies all 3 independence parameters
IV&V is vested in an organization that is separate from the development organization.
Close working relationships is required to ensure findings and recommendations are integrated rapidly back into the development process
Generally required for high integrity levels (4)
Loss of life, loss of mission, significant social or financial loss.
11/16/2018

11. Forms of Independence 2. Modified IV&V
System prime integrator is selected to manage system development including IV&V
Acquirer reduces acquisition time by passing responsibility to prime integrator
Managerial independence is compromised
IV&V effort reports to prime integrator
Technical independence is preserved: independent staff
Financial independence is preserved: separate budget
Appropriate for integrity level (3)
important mission and purpose
11/16/2018

12. Forms of Independence Integrated IV&V
IV&V performed by an organization that is financially and managerially independent from the development organization
IV&V organization is working side-by-side with the development organization
Impacts the technical independence counterbalance by interdependence
i.e. successes is tied to working together in a cooperative fashion.
Provide rapid feedback of V&V results
11/16/2018

13. Forms of Independence 4. Internal IV&V
IV&V is performed by personnel within development organization
Managerial independence is compromised:
IV&V effort uses same corporate procedures as development
Peer pressure from development may influence IV&V
Technical independence is compromised: vulnerable to overlooking errors by using same environment used by developers
Financial independence is compromised because development controls IV&V budget
Benefit: access to staff who know the system and software
5. Embedded V&V
Similar to Internal IV&V
Focus is on ensuring compliance to procedures and processes
Allows rapid feedback of V&V results into development process
11/16/2018

14. Elements of a Complete IV & V Effort
1) Review Developer’s methodologies
2) Requirements identification & allocation
3) Test matrix development
4) Verification tests
5) Validation tests
6) Quality control
7) Configuration management
8) Data management
9) Code Execution
Source: Jose Martin, Mike Mcgee, Dean Barten, ‘Independent Verification and Validation’.
11/16/2018

15. Allocation of IV&V Effort
Most intense at beginning and end of development cycle
Mid-part of development cycle: review process
Typical allocation of effort for a software development project *
Activity
Percentage of effort
Requirements analysis
12-17%
Design analysis
14-18%
Code analysis
18-22 %
Testing
18-26%
Tool preparation
2-15%
Management and reporting
15-25%
* Source: Deutsch, M., Willis, R., ‘Software Quality Engineering- A Total Technical and Management Approach’, Prentice Hall, 1988.
11/16/2018

16. IV&V at NASA
NASA determined the need for software IV&V after evaluating the causes of recent mission failures.
These were due, in part, to software issues that should have been identified during development or testing.
“Amount” of risk incurred by a project had to be assessed
To balance cost against potential benefits
Criteria determining when to consider IV&V
Using probability and consequence.
Consequences of failure were classified as:
Grave, Substantial, Marginal, and Insignificant.
Grave
Potential for loss of life – Yes
Potential for loss of equipment – Greater than $100,000,000
Potential for waste of resource investment – Greater than 200 work-years on software
Potential for adverse visibility – International
If any of the conditions are met, the software is considered to reside in that category.
11/16/2018
Rosenberg, L., Verification and Validation Implementation at NASA, Crosstalk, May 2001.

17. IV&V at NASA Criteria determining when to consider IV&V
Using probability and consequence.
Probability is estimated using 9 factors:
Software team complexity, contractor support, organization complexity, schedule pressure, process maturity of software provider, degree of innovation, level of integration, requirement maturity, and software lines of code.
Five risk categories within each factor were identified
Values 1, 2, 4, 8, and 16 were assigned to each category.
A weighting factor of 1 or 2 was identified for each factor.
11/16/2018
Rosenberg, L., Verification and Validation Implementation at NASA, Crosstalk, May 2001.

18. IV&V at NASA
11/16/2018

19. IV&V at NASA Criteria Application Example
Software team complexity – “up to 20 people at one location” = 4 * 2 = 8
Contractor support – “with minor tasks” = 2 * 2 = 4
Organizational complexity – “two locations but with same reporting chain” = 2 * 1 = 2
Schedule pressure – “non-negotiable” = 16 * 2 = 32
Process maturity – “ CMM Level 1 but with a successful history” = 8 * 2 = 16
Innovation – between proven but new and cutting edge = 8 * 1 = 8
Integration – almost stand alone = 2 * 2 = 4
Requirement maturity – “preliminary objectives” = 8 * 2 = 16
Lines of code = ~ 300K = 2 * 2 = 4
TOTAL = = 94
11/16/2018
Rosenberg, L., Verification and Validation Implementation at NASA, Crosstalk, May 2001.

20. IV&V at NASA Dark region
Where software consequences, likelihood of failure, or both are high. Projects having software that falls into this high-risk area shall undergo IV&V
Gray region
Represent projects with intermediate risk. Projects having software that falls into these areas shall undergo an evaluation to determine if IV&V is warranted.
11/16/2018

21. IV&V at NASA Results In the command-and-control system
IV&V identified design flaws that, if not corrected, would have resulted in a catastrophic hazard.
In the manned-space flight domain,
More than 4,000 problems were identified, 10 of the highest criticality, those that could result in loss of mission or loss of life.
For experimental flight vehicles
IV&V identified more than 300 requirements and design problems.
For ground systems
More than 250 legacy system requirements and mitigation problems were identified.
11/16/2018
Rosenberg, L., Verification and Validation Implementation at NASA, Crosstalk, May 2001.

22. Severity Definitions Five Year Implementation Plan, 2007-2012
11/16/2018
Five Year Implementation Plan,

23. In 2006, NASA allocated $29 million to the NASA IV&V Program budget.
11/16/2018

24. Types of IV&V Involvement
Full, In-Phase IV& V.
Most comprehensive of all possible IV&V efforts
Performed in parallel with the system development
Spans as much of the life cycle as possible.
IV & V must start no later than the requirements phase of full-scale development and,
it must continue long enough to include acceptance testing of the system.
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
James D Arthur, IV&V Course Notes, Virginia Polytech Institute and State University.
11/16/2018

25. Types of IV&V Involvement
2. Partial IV&V.
IV&V begins after the requirements phase has been completed.
The influence the IV&V group could and would otherwise have on the establishment and verification of requirements becomes an after-the-fact retrospective activity.
Must ‘go-back’ to review documents
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

26. Types of IV&V Involvement
3. Endgame IV&V
Must ‘look back’ in order to understand what happened
Added test and integration resources
Concentrate on the results of testing
Highly validation-oriented.
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

27. Types of IV&V Involvement
4. Audit-Level IV & V.
To "audit" plans, procedures, practices, and emerging products for adequacy, correctness, compliance to standards, etc.
Many times it is done as a quick fix of a program that is beginning to develop symptoms that something is going wrong.
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

28. Percentage of IV&V Cost to Total Software Development Cost%
Size in KSLOC
11/16/2018
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:

29. IV&V Costs and Savings
IV&V effort starting at the requirements phase and continuing through deployment would increase the development costs approximately 10% to 18%.
A significant portion of the costs of IV&V can be recouped.
When examining development efforts that initiate IV&V at the beginning of the coding phase
20% - 28% of the cost of IV&V is saved.
When examining development efforts that initiate IV&V at the beginning of the requirements phase
Savings of 92% - 180% of the costs of IV&V
50-50 Rule
Wise not to commit more than 50% at the beginning
Reserve a portion for unplanned, unforeseen activities
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

30. Methodology 5 Key Steps Step 1: Review all pertinent documents:
- SW requirements specifications methodology
- SW design methodology
- SW coding standards
- CASE tool documentation
- Test plan and procedure methodology
- SW quality control methodology
- SW configuration management methodology
11/16/2018

31. Methodology (continued)
Step 2: Interview to confirm the intended methodologies
Step 3: Analyze the methodologies individually & whole
- Each methodology supports the needs of its phase
- Each methodology supports the needs of the program management
- Data and document configuration control are an integral part of the development cycle
- Quality assurance is an integral part of the development cycle
- Methodologies as a whole support cost-effective software development and program management
11/16/2018

32. Methodology (continued)
Step 4: Develop a report on the results of the analysis including recommendations for change
Step 5: Support all discussions and corrective actions undertaken to improve the software development methodologies
11/16/2018

33. Differences and Advantages of IV&V Compared to SQA
SQA is an internal ‘watchdog’
The development organization’s objective of realizing maximum profit compromises concerns for quality. As such, decisions are often made that adversely impact the quality of the product being developed.
The resolution of internal political issues often overrides technical concerns about the quality of the product or process.
Because the SQA group is part of the development organization, it has very little influence in advancing the consideration of quality when confronted by pressures of schedule and cost.
IV&V is an external watchdog
Operates independently from the development organization
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

34. IV&V and CMM Maturity Levels 1 and 2
CMM Level 1
Crisis management mode
Developers and management hardly know where they are or where they are going.
IV&V can hardly plan its activities
CMM Level 2
Repeatable and flaky process
IV&V is more effective
Most effort goes to management related issues
Determining project status, auditing CM records, tracking.
25% of effort spent on non-technical issues
11/16/2018

35. Issues for Customer
How early should the IV& V contractor be brought on board,
How can the development contract and IV&V contract requirements be coordinated in both procurements to
Minimize cost and duplication of effort
Maximize usefulness of data, support systems, CASE tools, and documentation
Will geographic locations of the development contractor and the IV&V contractor have an any effect on the procurement
Can small-business contractors compete and perform effectively as IV&V contractors
11/16/2018
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:

36. Issues for Customer
5. How large should the IV&V effort be in respect to total development effort
6. What will the IV&V contractor develop and supply as part of the effort and what can be government-furnished
7. What sort of products can be expected from the IV&V effort
8. Will there be any problem with data rights between the IV&V contractor and the development contractor
11/16/2018

37. Issues for Customer
9. How can the IV&V contractor augment the customer's staff as an added resource
Should this be considered a good or bad practice?
10. How tightly locked in should the IV&V contractor's statement of work be
11. What percentage of IV&V's resources should be allocated against pre-defined tasks
12. What criteria are needed to evaluate IV & V proposals to the maximum advantage of the procurer
11/16/2018

38. Issues in IV&V SOW
6. Ensure data rights on behalf of the procurer for all items developed under the contract.
Ensure that test data are available from the contractor immediately following designated key-event tests.
Ensure that the IV&V can specify the extra data recording, etc. that is needed during contractor tests that are to be monitored.
Ensure that the IV & V will have the proper role in configuration management and CCB actions of both development contractor and customer.
Ensure that the IV&V will have adequate interface with the contractor's quality-assurance organization.
11/16/2018

39. IV &V Selection Issues Competency
Does the IV&V have the ability to do the job
Independence
Can the IV&V provide demonstrable independence
Technical Assessment
Does the IV&V have a sound assessment strategy
e.g. plan, evidence to be collected, etc.
                                      
Source: Yellow book, Application Note 4 – Independent Safety Assessment, May 2003.
11/16/2018

40. Example of Selection Criteria
Significant experience in the application domain
Significant experience in engineering processes
System and software design, production, testing, integration, etc.
Significant experience in V&V techniques, tools.
Significant experience of process assurance (e.g. safety audits)
Significant experience in utilization of Standards and procedures
e.g. Safety Standard Cenelec 50128
Technical, Managerial and Financial independence
From System Integrator
Should not be a threat to development organization
Should not be a competitor of the development organization
Should not be a supplier/partner of the development organization
Potential for conflict of interest
Experience in dealing with multiple ‘viewpoints’
customer, operators, maintainers, system integrator, suppliers.
11/16/2018

41. Proposal Evaluation
Methodology. A sensible and cost-effective approach to:
Requirements analysis and tracing
Design analysis (including algorithms)
Code analysis and independent verification via tools
Validation using complementary testing and analysis aids to those used by the software contractor
Tools and Analysis Aids
Existing repertoire of models and simulations applicable to the system and technology involved
Cost-effective mix of manual and automated analysis aids
Well-presented development approach and rationale for any new tools or aids required
Facilities
Company-owned or priority access to appropriate computer facilities
Well-designed testbed with total capabilities to execute code and analyze results
Maximum and effective use of government-furnished facilities
11/16/2018

42. Proposal Evaluation Staffing and Organization
Staff experience on the type of system being developed
Ability to start immediately upon award of contract
Organization that matches job and complements customer's organization
Single-point interface (especially important if subcontractor is involved)
Willing to support difficult geographic constraints
Experience
Staff experience in previous IV & V efforts
Staff experience in type of system, algorithms, computer language, etc.
Company experience in type of system involved, with proven track record of on-time, on-budget performance
Managerial and high-level corporate commitment.
11/16/2018

43. Working with IV & V Keeping IV&V aware of project progress and plans
Projects activities and meetings
Opportunity to attend, as an observer
Providing project documentation
Give early draft for comments
Managing of IV&V
Holding regular progress meetings
Monitoring independence
Project changes may impact independence
e.g. new contractor comes on board
Monitoring competence
Change to project may require additional competence
Providing advice
IV&V do not provide advice to project being assessed
11/16/2018

44. Summary of Benefits of IV&V
Improved software and system performance
Early detection of errors
Life cycle cost savings
Easier maintenance throughout the operational life of the system
Much higher level of user satisfaction
Reduced reaction time and better evaluation of proposed changes
Systems that don't break when stressed
Reduced development risk
Improved documentation
Improved communication and project visibility.
Source: Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
11/16/2018

45. References
IEEE Standard for Software Verification and Validation, IEEE Computer Society, IEEE Std 1012–1998.
Deutsch, M., Willis, R., ‘Software Quality Engineering- A Total Technical and Management Approach’, Prentice Hall, 1988.
Yellow book, Application Note 4 – Independent Safety Assessment, May 2003.
Robert O. Lewis, "Independent Verification and Validation : A Life Cycle Engineering Process for Quality Software". Interscience, ISBN:
James D. Arthur, Richard E. Nance, ‘Verification And Validation Without Independence: A Recipe For Failure’, 2001.
11/16/2018