1. Copyright 1999-2010 USC-CSSE 1 Quality Management – Lessons of COQUALMO (COnstructive QUALity MOdel) A Software Defect Density Prediction Model AWBrown and Sunita Chulani, Ph.D. Presented by Marilee Wheaton

2. Copyright 1999-2010 USC-CSSE 2 Outline Behavioral Underpinnings Hidden Factory Defect Types COQUALMO Framework The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

3. Copyright 1999-2010 USC-CSSE 3 USC Modeling Methodology

4. Copyright 1999-2010 USC-CSSE 4 Delphi Assessment Ask each expert the range for driver –Apply personal experience, –Look at completed projects, –Guess (WAG), Collect and share in a meeting: discuss why/how different people made their estimate Repeat until no changes Final values (for each parameter) Max=(Hmax + 4*AVEmax + Lmax)/6

5. Copyright 1999-2010 USC-CSSE 5 Adding Project Data Effort Adjustment Multipliers (typical) –Linear Regression

6. Copyright 1999-2010 USC-CSSE 6 Fig 11., Pg 170 SwCEwCII a posteiori Baysian Update Used to combine expert judgement with sampled data: spread of datasets weights combination

7. Copyright 1999-2010 USC-CSSE 7 Outline Model Framework *The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

8. Copyright 1999-2010 USC-CSSE 8 COQUALMO Model Framework

9. Copyright 1999-2010 USC-CSSE 9 Software Devel. Process(es) (cont.) : the hidden factory

10. Copyright 1999-2010 USC-CSSE 10 Outline Model Framework *The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

11. Copyright 1999-2010 USC-CSSE 11 The Defect Introduction (DI) Sub-Model Software Size estimate Software product, process, computer and personnel attributes (subset of COCOMO II factors) Defect Introduction Sub-Model Number of non-trivial requirements, design and code defects introduced

12. Copyright 1999-2010 USC-CSSE 12 A-Priori Expert-Judgment Based Code DI Ranges

13. Copyright 1999-2010 USC-CSSE 13 DI Model Equations For each artifact j, Quality Adjustment Factor (QAF) Estimated Number of Defects Introduced =

14. Copyright 1999-2010 USC-CSSE 14 Initial Data Analysis on the DI Model DIR = Defect Introduction Rate

15. Copyright 1999-2010 USC-CSSE 15 Outline Model Framework The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results *The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) COQUALMO Integrated with COCOMO II

16. Copyright 1999-2010 USC-CSSE 16 Defect Removal Sub-Model Defect removal activity levels Number of non-trivial requirements, design and coding defects introduced Number of residual defects/ unit of size Software Size Estimate The Defect Removal (DR) Sub-Model

17. Copyright 1999-2010 USC-CSSE 17 Defect Removal Profiles 3 relatively orthogonal profiles –Automated Analysis –People Reviews –Execution Testing and Tools Each profile has 6 levels –Very Low, Low, Nominal, High, Very High, Extra High Very Low--removes the least number of defects Extra High--removes the most defects

18. Copyright 1999-2010 USC-CSSE 18 Automated Analysis

19. Copyright 1999-2010 USC-CSSE 19 Static [Module-Level Code] Analysis "Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software (analysis performed on executing programs is known as dynamic analysis). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code. "*softwaredynamic analysissource codeobject code * http://en.wikipedia.org/wiki/Static_code_analysis

20. Copyright 1999-2010 USC-CSSE 20 Static [Module-Level Code] Analysis SWEBOK [sans references] "4.2. Quality Analysis and Evaluation Techniques Various tools and techniques can help ensure a software design’s quality. "Software design reviews : informal or semiformal, often group-based, techniques to verify and ensure the quality of design artifacts (for example, architecture reviews, design reviews and inspections, scenario-based techniques, requirements tracing) "Static analysis : formal or semiformal static (nonexecutable) analysis that can be used to evaluate a design (for example, fault-tree analysis or automated cross-checking) "Simulation and prototyping : dynamic techniques to evaluate a design (for example, performance simulation or feasibility prototype)" * Software Engineering Body of Knowledge. Alain Abran, et al., Swebok_Ironman_June_23_ 2004.pdf, pg 54

21. Copyright 1999-2010 USC-CSSE 21 Peer Reviews

22. Copyright 1999-2010 USC-CSSE 22 Syntactic Versus Semantic Checking Both sentences below are semantically correct, only one is semantically correct. A panda enters the bar, eats shoots and leaves. A panda enters the bar, eats, shoots and leaves.

23. Copyright 1999-2010 USC-CSSE 23 Execution Testing and Tools

24. Copyright 1999-2010 USC-CSSE 24 Technique Selection Guidance “Under specified conditions, …” Peer reviews are more effective than functional testing for faults of omission and incorrect specification (UMD, USC) Functional testing is more effective than reviews for faults concerning numerical approximations and control flow (UMD, USC)

25. Copyright 1999-2010 USC-CSSE 25 Residual Defects Equation Estimated Number of Residual Defects DRes Est, j = Estimated No. of Residual Defects for the j th artifact C j = Calibration Constant for the j th artifact DI Est, j = Estimated No. of Defects Introduced for the j th artifact (output of DI Sub-Model) i = Defect Removal profile DRF ij = Defect Removal Fraction 3 i=1

26. Copyright 1999-2010 USC-CSSE 26 Defect Densities from Expert- Judgment Calibrated COQUALMO

27. Copyright 1999-2010 USC-CSSE 27 Validation of Defect Densities Average defect density using Jones’ data weighted by CMM maturity level distribution of 542 organizations is 13.9 defects/kSLOC Average defect density using COQUALMO is 14.3 defects/kSLOC Residual defect density from Jones’ data (.135*1.4 +.196*7.5 +.669*18.3) = 13.9 defects/kSLOC

28. Copyright 1999-2010 USC-CSSE 28 An Independent Validation Study Aim: To validate expert-determined COQUALMO has correct trends in defect rates Sample Project: Size = 110.5 kSLOC Actual Defect Density = 6 defects/kSLOC

29. Copyright 1999-2010 USC-CSSE 29 Outline Model Framework The Defect Introduction Sub-Model –Expert-Judgment Model + Some Initial Data Results The Defect Removal Sub-Model –Expert-Judgment Model (Result of COQUALMO Workshop) *COQUALMO Integrated with COCOMO II

30. Copyright 1999-2010 USC-CSSE 30 Integrated COQUALMO