1. Software Defect Modeling at JPL John N. Spagnuolo Jr. and John D. Powell 19th International Forum on COCOMO and Software Cost Modeling 10/27/2004

2. John N. Spagnuolo Jr. & John D. Powell 2 Measurement & Benchmarking Process & Product Definition Capture, define, and refine repeatable processes and a set of engineering practices for project use Provide measurement infrastructure for projects, conduct empirical analysis, and package experiences for future use Software Engineering Technology Infusion Deployment Identify, evaluate, and support software tools and techniques to facilitate process and product improvement Infuse practices into project use; provide training, products, mentoring and consulting for projects SQI Project Engineering Provide overall technical infrastructure and work element integration SQI Project Engineering Provide overall technical infrastructure and work element integration The JPL SQI Project

3. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 3 SQI Measurement & Benchmarking The objective of the SQI Measurement Program is to provide the basis for a quantitatively based software management approach –Define models and measures –Create an infrastructure –Provide consulting and support –Produce Handbooks & Training Cost estimation and planning –Help develop total cost, schedule and plan for project activities and phases Quality planning and assessment –Help predict and assess the quality of products Management tracking –Help managers plan / monitor activities and assess risks during project execution Guiding improvement –Help JPL assess the overall effectiveness of software processes

4. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 4 Quality Purpose and Goals The purpose of this work is to predict software quality via the development of Defect Prediction Models –Predict defects and their effects on software projects early in project’s life cycle The current goal of this work is to make use of analysis of JPL defect data to support decision making. –Determine and make use of existing software defect trends –Determine the driving forces behind exceptions to trends Critical Discriminators (CDs) are the distinguishing characteristics of a project(s) that capture/quantify the driving force behind a deviation from over all JPL defect trends Combine Critical discriminators along with trends to assess threats to software project goals –Provide a means for managers to make use of Trends and CDs for Planning Prediction Corrective Actions Process Improvement

5. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 5 Project-Defect-SLOC-Chart

6. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 6 Early Defect Measurement Average Defect Density for –Flight Software (FSW) –Ground Software (GSW) –FSW & GSW Plot Lines with Slopes = Densities

7. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 7 One Perspective of Data: Average Work Hours per SDS vs SLOC Formed Two Linear Trends Philosophical Connection with Subsequent Charts

8. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 8 Preview of Major Findings New Trends: Given the basic Defect prediction Approach (Previous slide), the actual JPL defect data shows defects trends that are: –Are Counter intuitive on first inspection –More complex on further inspection New CDs: Newly developed software size CDs simplify the complex and counterintuitive Defect versus Size trend. –Managers may quickly make use of simple valuable sub-trends that apply to their projects: Without dealing with overall trend complexities that may be based on factors that do not apply to their project Based on characteristics such as size that may be estimated early in the software lifecycle and easily revised for accuracy as the project progresses. –The new CD provides a means for managers to make use of the trends for planning, prediction, corrective actions and process improvement.

9. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 9 The High Level Story Difference between Fault and Failure –Fault : error which may or may not have been discovered –Failure : Outward sign of an error (Discovered Fault) –In This presentation Failure = Defect Related Literature suggests that faults vs. code size may fit to an exponential curve Relation of exponential curve to Project G4 Data Plots –Linear fit is reasonable for Project G4 defects vs. size –Not necessarily a contradiction All faults may not have to be fixed More testing resources and/or time Project G4 defect curve ~ exponential fault curve Expand and correlate Project G4 analysis to all FSW and GSW Data collected thus far

10. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 10 Project G4 and JPL Projects Project G4 DATA : Number of Defects vs. SDS SLOC –Project G4 TEST (SDS’s) - Quadratic Fit the best –Project G4 OPS (SDS’s) - Quadratic Fit is the best –Project G4 TEST & OPS (SDS’s) - Quadratic Fit is the best –Project G4 TEST + OPS (SDS’s) - Quadratic Fit is the best JPL Projects: Number of Defects vs. Project Size –All FSW - Quadratic fit is the best –All GSW - Quadratic Fit is the best –All FSW & GSW - Quadratic Fit is the best

11. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 11 Interesting Fit for ALL JPL FSW

12. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 12 Curve Extends to GSW, GSW&FSW Project G4 TEST, OPS, TEST & OPS,TEST+OPS

13. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 13 Trends and Potential Explanation Upper bound on amount of testing resources & time –Testing resources / KSLOC is higher for smaller modules than for larger modules. –One possible explanation of defects / size ratios –Smaller Modules As code gets bigger more people tend to test more Defect curve increases to a certain point –Medium Modules With further size increases, testing resource approach their upper limit Defect curve begins to “level off” –Larger Modules Upper bound on testing resources for Project G4 Testing resources/KSLOC not big enough to maintain defect / size ratio observed in smaller modules Eventually defects / KSLOC trend curve begins to decrease

14. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 14 Where Does the Exponential come in? All FSW and GSW: Exponential Fit for LKSLOC < 247

15. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 15 Exponential Fits for JPL FSW, GSW Project G4 GSW TEST and Project G4 GSW OPS

16. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 16 Examination of Lower SLOC range: Project G4 –Computed (max LKSLOC - min LKSLOC)/3 –Project G4 TEST- Excellent fit to Exponential Curve < 103 LKSLOC –Project G4 OPS - Excellent fit to Exponential Curve < 57 LKSLOC –Project G4 TEST & OPS - Excellent fit to Exponential Curve < 103 LKSLOC –Project G4 TEST + OPS - Excellent fit to Exponential Curve < 103 LKSLOC Examination of Lower SLOC range: FSW, GSW, FSW & GSW –Computed (max LKSLOC - min LKSLOC)/3 –All FSW - Excellent fit to Exponential Curve < 118 LKSLOC –All GSW - Excellent fit to Exponential Curve < 247 LKSLOC –All FSW and GSW - Excellent fit to Exponential Curve < 247 LKSLOC Relevance to Exponential Curve

17. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 17 Observations at this Point The Empirical Evidence thus far seems to suggest: –Number of defects seems to increase as code size increases, peaks out, then begins a decreasing trend. –The exponential relationship between number of faults vs. code size, as indicated by the “Exponential curve”, appears to be most closely approximated by the relationship between the number of defects vs. code size for smaller sized Projects/SDS’s here at JPL –Code does not require exponential discovery of faults to work appropriately for JPL purposes

18. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 18 Summary Defect Differences from Test to Operations by SDS The Size Critical Discriminator (CD) for Defect Rates –Size CD is a code size threshold Represents successful development of a new and valuable CD Distinguishes between very different expected defects rate behaviors Allows defect prediction superior to simple “defects per lines of code” models –Significance of the Size CD Intuitive results below Size CD threshold Counterintuitive results above the Size CD threshold –Plausible and Reasonable explanations –Valuable rules for projects –Calculation of the Size CD and Differing Trends Analysis of Size CD and Defect Trends gives reasonable confidence level Very High Statistical Confidence Levels (R 2, etc..) but more data is need to make claims of high confidence overall Actual Data driven results Trends persist –Across multiple analyses of the data –Across various subsets of the data

19. 10/27/2004John N. Spagnuolo Jr. & John D. Powell 19 Relations between criticality 1 and criticality 2 defects For each “Number of PFR’s vs.. size” chart, construct a chart corresponding to defect density vs.. size – determine corresponding hump in trend curves Identify and Classify More Critical Discriminators –New individual CDs –Interaction of multiple CDs –For each CD associate modules, characteristics and data –Establish Trends for all modules corresponding to each CD Investigate interaction of multiple CDs at work in a given project / SDS Future Work