Estimating the number of components with defects post-release that showed no defects in testing C. Stringfellow A. Andrews C. Wohlin H. Peterson Jeremy Mange

Motivation For nearly any product, defects will appear after release The defects that were not detected in testing point to problems in the test process We would like to know how many of these post-release defects to expect

Motivation Using these estimates, release / re-test decisions can be made The authors wish to compare methods of post-release defect estimation

Defect Estimation The paper compares three methods of this type of estimation Experience-based Capture-recapture Curve-fitting Experience-based requires historical data, Capture-Recapture and Curve-fitting do not

Experience-based Estimation Requires some sort of historical data: Defect data Change data Historical data can be from previous products or prior releases of the same product If from previous products, they must be similar for these estimations to work

Experience-based Estimation Defect data Number of faults in a module can be estimated using the defect history of that module Change data Number of faults in a module can be estimated using the number of file changes Recent changes can be weighted more heavily

Capture-recapture Estimation A type of model originally designed for reviews and inspections Compares the number and types of defects found from multiple test sites Applies statistical estimation to the data based on assumptions about test sites

Capture-recapture Estimation Differences between test sites are accounted for in two areas: Ability to find defects Probabilities of finding specific defects This yields four models: Same ability, same probabilities Same ability, different probabilities Different ability, same probabilities Different ability, different probabilities

Capture-recapture Estimation

Curve-fitting Estimation Fits a mathematical curve to the data points to estimated remaining defects Two basic types: Decreasing – plot number of test sites that found each defect, sorted in decreasing order Increasing – plot cumulative number of defects found by each testing event

Curve-fitting Estimation With both increasing and decreasing models, the fitted curve is used to predict the number of remaining defects Both exponential and linear prediction models exist for each type

Approach 1. Collect the data For each discovered defect, count the number of test sites that discovered it

Approach 2. Apply non-historical methods Both capture-recapture and curve-fitting models are applied These provide estimates of the number of total defects

Approach 3. Apply experience-based method Use historical data to estimate the number of post-release defects For each past project (or release), calculate:

Approach 4. Using the calculations, estimate post- release defects For non-historical models, subtract the number of defects found in testing from the calculated total defects estimate

Approach 5. Compare this estimate to a decision threshold value If the number of expected post-release defects is acceptable, the product can be released If not, further testing must be performed Of course, the actual release decision will be based on many criteria, this is just one of those

Results This approach was carried out in a case study on a medical record system Results: Capture-recapture and curve-fitting: 5%-20% error Experience-based: 3%-5% error

Conclusion If data for similar projects is available, experience-based models provide the best estimations for post-release defects However, non-historical (capture-recapture and curve-fitting) models also provide fairly accurate estimates independent of past data Post-release defect estimates should be used in release decisions

Questions?