1. Civil Government Services Group 1 Return on Investment of Independent Verification and Validation: Indirect Benefits James B. Dabney, Gary Barber, Don Ohi, Kurt Woodham Software Assurance Symposium 29 July 2003

2. Civil Government Services Group 2 Overview  Approach  Overall Results  Top Four Benefits  Conclusions

3. Civil Government Services Group 3 Study Approach  Benefits of Value  Quantifiable Characteristics  Quantification Methods

4. Civil Government Services Group 4 Benefits of Value  Identify as many benefits as possible  Determine whether each benefit can be substantiated by tangible data  Assess quantifiability of each benefit  Identify quantification  Published data  IV&V and project artifacts

5. Civil Government Services Group 5 Quantifiable Characteristics  Score benefits  Develop method to convert quantifiable benefits into consistent measure (EPM, dollars)

6. Civil Government Services Group 6 Quantification Methods  Identify quantification procedure for each benefit  Unique method for each benefit  Refine and test

7. Civil Government Services Group 7 Results – Candidate Benefit List  Identified 84 candidate benefits  Refined list to 26 quantifiable benefits  Scored (High=3, Medium = 2, Low = 1)  Universality (weight = 1)  Practicality (weight = 3)  Potential value (weight = 4)  Credibility (weight = 5)

8. Civil Government Services Group 8 Top Four Indirect Benefits  Improved testing  Reduced high-criticality errors  Requirements clarification  Reduced leakage to operations

9. Civil Government Services Group 9 Improved Testing  Direct benefit is correction of test case deficiency  Indirect benefit is error identified as consequence of corrected test case  Computing return  Identify IV&V issues resulting in test case change  Correlate issues with defects found in testing  Incorporate results into direct ROI computation  Apply high-criticality and phase leakage methodology where appropriate

10. Civil Government Services Group 10 Improved Testing Example  Mission-critical ground software system  IV&V identified 61 test deficiencies  One added test case identified flaw in redundancy management state machine  Flaw would have prevented detection of serious system fault  Direct ROI return for this error is small, indirect large using high-criticality methodology

11. Civil Government Services Group 11 Reduced High-Criticality Errors  CARA focuses IV&V on high-criticality portions of system  Removing high-criticality defects has benefits beyond cost-to-fix (direct ROI)  Value of mission  Value of vehicle  Crew and control center time during mission  Compute using expected value based on estimated probabilities of occurrence

12. Civil Government Services Group 12 Reduced High-Criticality Errors Example  Incorrect handling of thermal control system failure  Disable primary GN&C computer  Fault not recognized – backup GN&C stays off line  Vehicle placed in uncontrolled state  Three undesirable operational outcomes  Loss of vehicle at cost of $40B, 0.00001 probability  Loss of mission at cost of $4B, 0.00001 probability  Failure to annunciate fault, $4M, 0.01 probability  Expected value = $480,000

13. Civil Government Services Group 13 Requirements Clarification  Requirements ambiguities  Wording open to misinterpretation  Conflicts between different representations of same requirement  Benefits of removing  Reduced development cost resulting from reduced probability of implementation errors  Reduced maintenance cost due to increase in understandability

14. Civil Government Services Group 14 Reduced Development Cost Computation  Increment to effective SLOC for requirements ambiguity found in requirements phase is where k w is function point weight, k L function point to SLOC factor, p i probability of error of type i resulting from requirements ambiguity, k s is SLOC reduction factor, d i is ambiguous requirement removal rate phase i

15. Civil Government Services Group 15 Reduced Maintenance Cost Computation  COCOMO-II reuse model includes term SU for understandability  Estimate increment to SU as  FP A is function points associated with ambiguous requirements  FP T total function points  Increment to ESLOC (effective SLOC due to reuse)

16. Civil Government Services Group 16 Reduced Error Leakage to Operations  Direct ROI model assumes no defect leaks to operations  Experience indicates a significant number of errors will manifest in integration testing and operations  Expand direct ROI leakage model to include integration and operations  Requires adjustment to direct ROI coefficients

17. Civil Government Services Group 17 Conclusions  Identified and rank-ordered 26 unique indirect benefits of IV&V  Analyzed top four benefits in detail  Improved testing  Reduced high-criticality errors  Requirements clarification  Reduced leakage to operations  Direct and indirect benefits form comprehensive ROI model and basis of predictive ROI model