A Study of Estimation Methods for Defect Estimation by Syed Waseem Haider and Dr. João W. Cangussu The University of Texas at Dallas

Outline Introduction Classical Approach Defect Estimators Conclusion & Future Work

Introduction The major goal of software testing is to find and fix as many defects as possible under given constraints so as to release a product with reasonable reliability. Time to achieve the established goal and percentage of the goal achieved up to the moment are important factors to determine the status. The clear view of the status of the testing process is crucial to find a trade-off between releasing a product earlier or investing more time on testing. Many defect prediction techniques have addressed this important problem by estimating the total number of defects. The availability of an accurate estimation of the number of defects at early stages of the testing process allows for proper planning of resource usage, estimation of completion time, and current status of the process. Also an estimate of the number of defects in the product by the time of release allows the inference of required customer support.

Introduction (contd.) We will discuss various estimation methods which are used to develop defect estimation techniques. We will discuss assumptions that each method makes about the data model. probability distribution, and mean and variance of data and estimator. statistical efficiency of the estimators developed from these estimation methods.

Estimation Approaches Some of the Bayesian defect estimators in the field are Bayesian Estimation of Defects based on Defect Decay Model (BayesED3M) A Bayesian Reliability Growth Model for Computer Software by B. Littlewood and J. Verrall Bayesian Extensions to Jelinski-Moranda Model by W. S. Jewell

Classical Approach For each method we will discuss Requirements How to develop estimator Statistical Performance Merits & Demerits

Basic ingredients Collection of data Data modeling Likelihood function

Collection of data samples Any estimation technique needs samples or data from the ongoing system testing process. Samples can be in the form of number of defects found each day or week or any other time unit. Samples can be total number of defects found by any instant of time. Examples of sampling data In reliability models number of defects discovered per execution time. Calendar time versions of reliability models also exist. Estimation of Defects based on Defect Decay Model (ED3M) works for both calendar time and execution time.

Data modeling Let be the parameter to be estimated. It is the total number of defects. A data model is used to relate to the data samples drawn from the system testing. Data model must also account for random behavior caused by work force relocation, noise in the testing process, testing of varying complexity product, among others. Lets assume that we take nth sample x[n] which contains corrupted by random noise w[n] as given by Obesrvations of made in N intervals is given by Note that in Eqs.1 and 2 is linearly related to data. In Eq.2 h is the observation vector. It can contain information such as number of testers, failure intensity rate, number of rediscovered faults for each sample, etc. A linear data model is recommended for two reasons first it is more likely to provide a closed form solution and secondly its more efficient as will be discussed later.

Likelihood function The joint probability distribution of data is given by p(x[0],x[1],…,x[N-1]; ) or in the vector form p(x; ) (PDF of data). p(x; ) is the function of both data x and the unknown parameter . For example if for a given data set is changed to the value of p(x; ) will change. When p(x; ) is seen as the function of it is called likelihood function. Intuitively p(x; ) provides how accurately we can estimate

Minimum Variance Unbiased (MVU) estimator

Minimum Variance Unbiased (MVU) estimator (contd.) Probability distribution of the data must be known. The problem of finding the estimator is simply to find a function of data. must be unbiased . Variablility of the estimates determines the efficiency of the estimator. Among several estimators one with the lowest variance is the efficient estimator. Among various methods to determine lower bound on the variance Cramer-Rao Lower Bound (CRLB) is easier to determine. Many defect estimators in the field assume the probability distributions for the data. e.g. Musa-Okumoto Model poisson process, (add more here )etc.

Minimum Variance Unbiased (MVU) estimator (contd.) CRLB states: it is assumed that the PDF p(x; ) satisfies the regularity condition where the expectation is taken with respect to p(x; ). Then the variance of any unbiased estimator must satisfy An estimator which is unbiased, satisfies the CRLB and is based on linear data model is called an efficient MVU estimator. It is found using Eq.5 The efficient MVU estimator and its variance is given by Eqs. 6 and 7 respectively In other words it efficiently uses the data to find the estimates.

Minimum Variance Unbiased (MVU) estimator (contd.) It may happen that we are able to find an estimator whose variance is less than other estimators but not less than CRLB. We will simply call such an estimator MVU estimator. In other fields such as signal processing and communication systems where the system or model under investigation is well defined in terms of physical constraints, it is possible to find efficient MVU estimator. In software engineering no model completely captures all the aspects of a software testing process. Different models are based on different assumptions and this lack of consistency hints towards the absence of a mature testing model. Therefore its unlikely to find an efficient MVU estimator

MVU estimator based on Sufficient Statistic It is possible that for a given data model CRLB cannot be achieved. In other words a solution similar to the one given by Eq.5 may not exist.

MVU estimator based on Sufficient Statistic (contd.) The minimal data required to make PDF of data p(x; ) independent of unknown parameter is called sufficient statistic. For example a simple estimator = x[n] will have high variance in estimating . But if sufficient data x[0],x[1],…,x[N-1] is available then new sample x[N] will not provide additional information about . If sufficient statistic exist then p(x; ) can be factorized as given by Eq.9 according to the Neyman-Fisher Factorization theorem. In Eq. 9 T(x) is sufficient statistic. A function of T(x) is an MVU estimator only if it is unbiased . As discussed earlier p(x;theta) is dependent on both data x and theta. Intuitively we can see that theta_hat will have lower bound on the variance as more new data is not contributing any new information about theta.

MVU estimator based on Sufficient Statistic (contd.) The PDF of ED3M the technique developed by authors can be factorized as given by Eq.9. The estimator of ED3M is an unbiased function of T(x) but we do not claim that estimator of ED3M is based on sufficient statistic for the following reason. An example of sufficient statistic is that we want to estimate the accuracy of a surgical precision laser. We take sufficient samples to estimate the average precision achieved as shown in the figure. In software testing as Dijkstra noted, testing shows the presence of defects but not their absence. Even though as time elapses rate of finding new defects subsides significantly but there will be new defects now and then. Overall testing process can be considered an increasing function of defects (in one not ‘around’). We can only forecast saturation of finding the defects. A change in strategy can result in sudden burst of more defects. Because of this behavior of testing process the notion of sufficient statistic in software testing is arguable. Therefore even though ED3M fulfills the mathematical requirements of a sufficient statistic estimator, we do not claim that its based on this method. It can be seen the samples are around the desired value. The key term here is ‘around’.

Maximum Likelihood Estimator (MLE) If we cannot find either sufficient statistic or for theta or an unbiased function of sufficient statistic then we resort to MLE.

Maximum Likelihood Estimator (MLE) (contd.) Many practical estimators developed are based on MLE. Important properties of MLE MLE is asymptotically (as ) an efficient estimator. For Linear data model as given by Eqs.1 and 2 MLE achieves CRLB for finite data set. If an efficient estimator exists MLE will produce it. The basic idea is to find the value of theta that maximizes ln p(x; ) the log-likelihood function for a given x. If a closed form solution does not exist a numerical method such as Newton-Raphson can be used to approximate the solution. Numerical approximation may not necessarily converge to maximization of ln p(x; ) to produce MLE. An example of numerical approximation of MLE is Musa-Okumoto model. Authors were able to find a closed form solution of MLE for ED3M. As pointed out in Data Modeling slide it is advantageous to have Linear data model.

Method of Moments If either MLE cannot be found (and its approximation is insufficient) or is computationally intensive then we can try Method of Moments. We must know the distribution of data.

Method of Moments (contd.) Method of moments is generally consistent. Given p(x;theta) if we know that the kth moment of x[n] is a function of as given by Eq.11. We approximate the kth moment of data x, by taking average of x(k) as given by Eq. 13. If f is an invertible function as given by Eq.12 then substitution of into Eq.12 results in the estimator as given by Eq.14.

Best Linear Unbiased Estimator (BLUE) If the probability distribution of the data is unknown or in other words distribution of noise is not known, but the first two moments are known we can develop BLUE.

Best Linear Unbiased Estimator (BLUE) (contd.) BLUE is based on two essential requirements called linearity conditions Data model is linear. Estimator itself is a linear function of data. The two linearity conditions are given by Eqs.2 and 15. Note that second linearity condition is necessary to make unbiased as given by Eq.16. BLUE is a suboptimal estimator because lower bound of its variance is unknown. It can be successfully used if its variance is in acceptable range and it is producing results with reasonable accuracy.

Best Linear Unbiased Estimator (BLUE) (contd.) A limitation of this method from practical point of view in software testing is that we have to know the variance of noise. In the present day no detailed study has been done which has investigated the statistical characteristics of noise in testing process. A simple way to approximate the variance of noise is to find the variance of data as given by Eqs.17 and 18. However the effects of this approximation on the performance of the BLUE estimator are unknown with respect to software testing.

Least Square Error (LSE) If either we don’t know the first two moments of noise or data, or the data model is not linear we cannot find BLUE. Then resort to LSE.

Least Square Error (LSE) (contd.) It is the most commonly used approximation or estimation method. The geometrical interpretation of LSE is more intuitive. If we have data points in space the LSE finds a curve which minimizes the distance from all these points together. A weakness of LSE is that it is sensitive to outliers (points which are away from the group of points). Due to these outliers the curve may be found away from the vicinity of points. A simple way to remedy this situation is to ignore the outliers from the data set. Main advantages of LSE is that its simple to develop and no information about the probability distribution of the data set or noise is needed. On the other hand the statistical performance of LSE is questionable. Authors have use LSE to approximate the values of and the defect decay parameters in ED3M . From the application of ED3M on several industrial data sets and simulation data sets the performance of LSE estimator for and was concluded acceptable.

Defect Estimators All of these approaches are based on MLE, where MLE requires assumption about probability distribution. Therefore each model defines distribution by making some assumptions while ignoring other facts. Hence it can be safely deduced that no model will work in all situations. Padberg’s Maximum likelihood estimates for the Hypergeometric software reliability model Musa-Okumoto logarithmic Poisson execution time model for software reliability measurement Estimation of Defects based on Defect Decay Model (ED3M) All estimators based on MLE

Padberg’s Approach Padberg showed that growth quotient Q(m) of the likelihood function L(m) when greater than 1 indicates that likelihood function is indeed increasing and provides maximum likelihood estimates. In Eq.19 m is the initial number of faults, wn is the number of newly discovered and rediscovered faults for the nth test and cn is cumulative number of faults in n tests. For a give data set cn first find x = cn+1 then Q(x). If Q(x) > 1 then set x = x+1 and fine Q(x) again. Keep repeating the steps until Q(x) ≤ 1. Statistical performance of Q(m) is not discussed. We do not know if the variance of Q(m) is asymptotically bounded by CRLB in other words if Q(m) is asymptotically an efficient MVU estimator. Even though the underlying data model is not known but it can be observed from Eq.19 that model is nonlinear. If the data model is nonlinear then it cannot achieve CRLB for finite data records.

Musa-Okumoto logarithmic poison execution time model Musa-Okumoto proposed a reliability model based on the assumption that the expected number of faults by time t are Poisson distributed as given in Eq.20. The parameters to be estimated are the initial failure intensity and the rate of reduction in the normalized failure intensity per failure. The data model in Eq.21 is a nonlinear function of and , hence MLE will not achieve CRLB for finite data set. A closed form solution of MLE could not be found for Eqs.20 and 21. Therefore a numerical approximation of MLE is needed. Whether the approximation of MLE will be asymptotically an efficient MVU estimator is not guaranteed.

Estimation of Defects based on Defect Decay Model (ED3M) The data model of ED3M is given by Eq.22, where D is the defect data vector, h is the observation vector and w is noise vector. Vectors are of dimension Nx1. We have assumed that D is normally distributed and the PDF of D is given by Eq.24. The initial number of defects in software is given by R0 The MLE estimator for R0 is given by Eq.25. As seen in Eq.22 the data model is linear, therefore MLE estimator in Eq.25 can achieve CRLB for finite data set and will be an efficient MVU estimator.

Comparison between ED3M and Padberg's MLE, where the total number of defects is 481 and the total time length is 111 units.

Comparison between ED3M and Musa-Okumoto model, where the total number of defects is 136 and the total time length is 88.682x103 CPU seconds.

Conclusion and future work An accurate prediction of total number of software defects helps in evaluation of the status of testing process. But the accuracy of the estimator owes to the estimation method which is used to develop the estimator. We have tried to provide a general framework of available estimation methods for researchers who are interested in defect estimation. Although discussion had been around software testing and defect estimation but its general enough to be used for other estimation problems. We have elicited the requirements of each method. We have also discussed the statistical efficiency that each method offers. Even though the discussion is limited to single parameter estimation, it can be easily extended to a vector of parameters to be estimated. In future we will extend our discussion to Bayesian Approaches and expand the analysis of existing estimators to be more comprehensive.