Calculation of Software Failure Probability and Test Case Selection February 14, 2007 Kim, Sung Ho

1 TABLE OF CONTENTS q Background q Software Failure Probability and Test Case Selection q Summary q Further Works

2 q Want to establish a quantitative software reliability model which is  straightforward, analytic and detail, easy to determine, valid even if there is little or no failure data, not complicated, realistic (no “not implemented requirements”)  Merits and demerits of the reliability prediction models proposed up to now Background Characteristics Models Data requiredMeritsDemerits MTTF Failure counts Failure times Straightforward Not valid if there is little or no failures Defect Density Software size Number of defects Analytic and detail Complicated analysis procedure (detailed inspections for phases and dynamic analysis) Function Points Types of dataAnalytic and detailComplicated analysis procedure Test Coverage Test coverage data Number of defects found Reflects the defects covered, defects remaining in the code, and tested number of statements Even if all the codes are covered by tests, the defect coverage is not perfect Requirements Traceability Number of requirements in the documents and codes Easy to determine The number of requirements not implemented in the code should be determined from RTM Bugs per lines of codes Number of modules Code size per module Easy to determine Empirical expressions Even a smallest module has many defects Different V&V levels generate different software quality  There is no model which considers the test inputs

3 q The defects and failures of a software  the defect in a software is invoked by input sets  the software will not fail if the defects are latent (not invoked by input sets)  the software can not be tested for all complete combinations of input sets  for most cases, exhaustive testing is not practical! (no perfect testing)  a large number of test cases should be used to guarantee some reliability level  the selection of test cases is closely related to the estimation of software reliability  the selection of test cases (test case coverage) can be a measure for software reliability Background a software Input set-1 Input set-3 Input set-2 software defect software fails at this point by input set-2

4 q Number of test cases [1]  system unavailability = 1.8 x Failures/Demand with no software failures availability = 1 – unavailability =  several tens of thousands of test cases are required to guarantee a given level of software reliability  combination of failure inducing test cases are not counted in this consideration  test cases are evenly and randomly selected without considering the operation profile Background Confidence Level (%) Reliability ,302 2,995 4,603 6,905 Number of Test Cases per Reliability and CL 100c : confidence level (%) r : reliability requirement number of test cases [1] J. H. Poore, Harlan D. Mills, and David Mutchler, “Planning and Certifying Software System Reliability”, IEEE Software, pp.88-99, Jan

5 q Binning the input domain for test case selection [2]  the most general form of an input distribution proposed by K. W. Miller  define a partition of the input domain of F, and decompose the domain of F into “bins” in such a way that each element in a bin is equally-likely to be drawn  each bin, say bin i, has an associated bin probability p i = n i p(x) [2] Keith W. Miller, et al., “Estimating the Probability of Failure When Testing Reveals No Failures”, IEEE Transactions on Software Engineering, Vol.18, No.1, pp.33-43, Jan., 1992 p i : bin probability n i : the number of elements in bin i p(x) : the common probability of selection for any single element in bin i Software Failure Probability and Test Case Selection Domain of function F i = 1i = 3i = 2 n 1 = 3n 3 = 4n 2 = 3 p 1 = 3/10p 3 = 4/10p 2 = 3/10 p(x) = 1/10 (equally-likely) Green points, test inputs revealing no failure Black points, test inputs revealing failure

6 q Assumptions for the binning the input domain  All elements in the domain must be represented  No element may appear in more than one bin  Estimation of software failure probability  The true software failure probability after testing for all elements where 1, for failure 0, for no failure and p(x) = 1/(# of elements) for equally-likely selection case  Estimation of software failure probability after t times of testing where a and b are constants of Beta( a,b ) distribution and determined by prior information about the software testing x f(x) a=2 b=2 a=5 b=3 10 Beta density function [2] Keith W. Miller, et al., “Estimating the Probability of Failure When Testing Reveals No Failures”, IEEE Transactions on Software Engineering, Vol.18, No.1, pp.33-43, Jan., 1992 Software Failure Probability and Test Case Selection

7 q Miller’s way of estimation for software failure probability  needs prior information about the software testing  needs to determine the beta constants, a and b, for next estimation of software failure probability q Practical software  has its own operating profiles which are not identical to the equally-likely test inputs  is not tested by, first, selecting one test case, then executing the test, and determining constants for analysis, and hence  is not, in many cases, appropriate to adapt Miller’s way Software Failure Probability and Test Case Selection

8 q Criteria for the selection of test cases  Operating profiles of the software  Combinations of highly probable input variables should be tested for several types of inputs T C : around 296°C for [ 230°C, 330°C ] inputs T H : around 327°C for [ 250°C, 350°C ] P : around 2250 psia for [ 1495 psia, 2489 psia ] TcTc 230 °C330 °C296 °C input range normal operating range Software Failure Probability and Test Case Selection

9 q Another way of determining probability density functions  determine the operating profiles first  the software is operated mainly in normal plant condition ranges  the failures to be invoked by the black point of normal range have high effect on the failure rate calculation  the black points in transient range have little effect on the failure rate calculation,  furthermore, the black points during plant shutdown has little effect on the system even if the software fails TcTc 230 °C330 °C296 °C f(T c )  this means that the selection should be distributed densely around the operating range, not over the whole range Software Failure Probability and Test Case Selection input range normal operating range Beta density function A black point

10 q Another way of determining probability density functions  adjust the beta density function to determine the constants, and, per variable considering the operating profile  then apply Miller’s estimator using the current selection probability where and are operating-adjusted beta constants for j th input Software Failure Probability and Test Case Selection TcTc 230 °C330 °C296 °C f(T c ),

11 Summary q most of the software reliability models, up to now, rarely incorporate test case selection method, q but the defects in a software are invoked only by inputs; otherwise, they are latent without inducing any software failures q and, in many cases, operating profiles are very important factor to test the software properly q Miller proposed to use beta functions to estimate the software failure probability if prior test information is known and the test is performed step by step q to avoid the nuisance of several analysis steps and to incorporate the features of operating profiles, the constants of beta distribution function can be determined first to estimate the software failure probability

12 Further Works q detailed ways to determine the beta function constants should be surveyed  adjusting beta functions, or  curve-fitting q a sample program can be used to confirm the estimator