1. West Virginia University Towards Practical Software Reliability Assessment for IV&V Projects B. Cukic, E. Gunel, H. Singh, V. Cortellessa Department of Computer Science and Electrical Engineering Department of Statistics West Virginia University September, 2001. Initiative Title: Integrating Formal Methods and Testing in a Quantitative Software Reliability Assessment Framework

2. West Virginia University Overview ä Introduction and Motivation ä Application example: DOLILU II ä Assessment of Software Reliability ä Statistical sampling ä Bayesian approach with ignorance priors ä Bayesian approach with non-ignorance priors ä Bayesian Hypothesis Testing ä Conclusions and Further Work

3. West Virginia University Introduction ä Quantification of the V&V activities is always desirable, but ä Is software reliability assessment practical for IV&V of safety/mission critical systems? ä Time and cost considerations may limit the appeal ä Reliability growth applicable only to integration testing, a tail end of IV&V. ä Estimation of operational usage profiles is rare ä Lifecycle V&V results ignored ä Opinions may limit the appeal too

4. West Virginia University Introduction ä Regulatory view: DO178-B (Software Considerations in Airborne Systems and Equipment Certification) “… methods for estimating the post-verification probabilities of software errors were examined. The goal was to develop numerical requirements for such probabilities for software in computer-based airborne systems of equipment. The conclusion reached, however, was that currently available methods do not provide results in which the confidence can be placed to the level required for this purpose... If the applicant proposes to use software reliability models for certification credit, rationale for the model should be included in the Plan for Software Aspects of Certification, and agreed with by the certification authority.”

5. West Virginia University Why impractical? ä Most verification and validation techniques are qualitative in nature. ä Typical approaches to software reliability estimation based exclusively on operational (system) testing. ä Neglects the investment made in other V&V techniques ä Requirements readings, inspections, problem reports and tracking, unit level tests… Req Design Code Test (Verification & Validation) Unit IntegrationAcceptance IV&V Implementation IV&V

6. West Virginia University Motivation ä Can software reliability assessment benefit from: ä Lifecycle long IV&V activities ä Qualitative (formal and informal) V&V methods? ä Can the amount of testing needed to assess mission critical reliability levels be reduced? ä Realistic case study

7. West Virginia University EXECUTIVEEXECUTIVE SHUTTLE ORBITER RANGE SAFETY Generate Wind Profile Range Data Guidance Cmds Tracking Data Wind Data Design Guidance Cmds Simulate Trajectory Verify Trajectory and Guidance Cmds DIVDT Generate Range Data Transmit Guidance Cmds and Range Data Verify Trajectory and Guidance Cmds Simulate Trajectory Integrated Day-Of- Launch I-Load Update (DOLILU) System Evaluate Results Decision Data Decision Data Case Study

8. West Virginia University DOLILU II Assessment Goals ä Failure probability under 10 -4 ä Due to the criticality of the program, required confidence level should surpass 0.99 ä Available methods for reliability estimation: ä Formal verification: virtually impossible ä Rigorous inspections, fault based and white box testing performed by an independent IV&V team ä Done, but observations were never quantified ä Reliability growth models cannot be used

9. West Virginia University Reliability Assessment Framework ä Random testing and Bayesian inference chosen for assessment ä Assessment must take into account failure free operational use of DOLILU II, and the results of performed V&V activities ä Bayesian inference ä Allows inclusion of a subjective probability of failure ä Subjective estimate based on observed behavior, reflects beliefs ä Hypothesis on the event occurrence probability is combined with new evidence, which may change the degree of belief ä In reliability assessment, Beta distribution is frequently used due to its mathematical flexibility and tractability ä Beta distributions form a conjugate family

10. West Virginia University Statistical Assessment (no assumptions)  P(    Required testing effort (N), from random sampling: ä Number of test cases as a function of the required failure rate, with C=0.99 Value of  Number of Tests 10 -2 458 10 -3 4,602 10 -4 46,048 10 -5 460,514 10 -6 4,605,167 Required testing effort not realistic.

11. West Virginia University Bayesian assessment (non-ignorance priors) ä DOLILU underwent extensive IV&V ä Partial correctness proofs, requirements & design readings, code inspections, rigorous development practices ä Sound formulation of prior beliefs is subject to further research ä Historical data on failure occurrences under the same IV&V regime ä Historical data on failure occurrence reduction following the application of the specific verification techniques ä Process effectiveness measures [Smidts 98] ä Represent the application of a specific verification method by an appropriate number of random tests [Miller 94]

12. West Virginia University Bayesian estimation (non-ignorance priors) ä Assume that we can say that the system has achieved desired reliability prior to certification testing. ä This “guess” should be “reasonably accurate” ä Use random tests (operational profile) to corroborate assumed system failure probability ä How many random tests U should be performed?

13. West Virginia University Benefits ä What if corroboration testing is not failure free? ä Keep adjusting the value of U [Littlewood 97]

14. West Virginia University Bayesian Hypothesis Testing (BHT) ä Problem of Bayesian estimation: ä Categorical assumption that the program meets required reliability must be made. ä We can put a probability on this assumption! ä Certification testing now searches for the evidence in favor of the hypothesized reliability  H o :   o alternative hypothesis

15. West Virginia University

16. Summary ä Bayesian framework for reliability assessment allows: ä Inclusion of IV&V activities into the reliability assessment. ä A significant reduction in the number of tests. ä Software reliability of DOLILU can be assessed with a reasonable effort. ä CAUTION: Do you trust your (I)V&V methods? ä Research directions ä Sound formulation of prior beliefs from IV&V. ä Can prior beliefs be based on the “formality” of the V&V methods (formal methods)? ä Inclusion of CRITICALITY and RISK parameters. ä Other case studies!!!