1. © USC-CSE 2001 Oct. 24. 20011 Constructive Quality Model – Orthogonal Defect Classification (COQUALMO-ODC) Model Keun Lee ( keunlee@sunset.usc.edu)keunlee@sunset.usc.edu & Sunita Chulani (Sunita_Chulani@us.ibm.com)

2. © USC-CSE 2001 Oct. 24. 20012 COQUALMO, ODC - Current shortfalls COQUALMO-ODC Model Objectives - Example of desired model Breakout Group Issues Outline

3. © USC-CSE 2001 Oct. 24. 20013 COCOMO II Current COQUALMO System COQUALMO Defect Introduction Model Defect Removal Model Software platform, Project, product and personnel attributes Software Size Estimate Defect removal profile levels Automation, Reviews, Testing Software development effort, cost and schedule estimate Number of residual defects Defect density per unit of size Defects Introduced Rqts. Design Code

4. © USC-CSE 2001 Oct. 24. 20014 - Nominal Defect Introduction Rates Delivered Defects / KSLOC Composite Defect Removal Rating COQUALMO Defect Removal Estimates

5. © USC-CSE 2001 Oct. 24. 20015 - Example : Code Defects; High Ratings Analysis : 0.7 of defects remaining Reviews : 0.4 of defects remaining Testing : 0.31 of defects remaining Together : (0.7)(0.4)(0.31) = 0.09 of defects remaining How valid is this? - All catch same defects : 0.31 of defects remaining - Mostly catch different defects : ~0.01 of defects remaining Need to determine which techniques find which defect type - Orthogonal Defect Classification (ODC) data can help Multiplicative Defect Removal Model

6. © USC-CSE 2001 Oct. 24. 20016 ODC A technology for software process measurement and analysis, providing a standard of unique, non-redundant attribute definitions for defect classification as well as analysis metrics and procedures for process evaluations Orthogonal Defect Classification (ODC)

7. © USC-CSE 2001 Oct. 24. 20017 Activity/Trigger How defect detected Impact Customer view Interaction Target/Type What was fixed Source Where defect located Qualifier Missing, Incorrect,.. Age History Feedback on Verification ProcessFeedback on Development Process Orthogonal Defect Classification (ODC)

8. © USC-CSE 2001 Oct. 24. 20018 - IBM Results (Chillarege, 1996) ODC Data Attractive for Extending COQUALMO

9. © USC-CSE 2001 Oct. 24. 20019 One-size-fits-all model - May not fit COTS/Web, embedded applications Defect uniformity, independence assumptions - Unvalidated hypotheses COCOMO II C-S-Q trade offs just to RELY levels - Not to delivered defect density, MTBF Need for more calibration data - ODC data could extend and strengthen model Current COQUALMO shortfalls

10. © USC-CSE 2001 Oct. 24. 200110 Outline Current COQUALMO Approach - Current shortfalls COQUALMO-ODC Model Objectives - Example of desired model Breakout Group Issues

11. © USC-CSE 2001 Oct. 24. 200111 Support cost-schedule-quality tradeoff analysis Provide reference for defect monitoring and control Evolve to cover all major classes of project - With different defect distributions (e.g. COTS-based) - Start simple;grow opportunistically COQUALMO-ODC Model Objectives

12. © USC-CSE 2001 Oct. 24. 200112 6 9 12 15 30 60 90 600K 1200K 1800K Cost @ $20K/PM Effort (PM) VL L N H VH Current: RELY rating Desired: Defect MTBF KSLOC (hr) 100 10 1 0.1 0.01 1 24 720 17,000 10 6 Details for any given point- next Time (Mo.) Example of Desired Model - I

13. © USC-CSE 2001 Oct. 24. 200113 30 KSLOC; RELY = VH; 75PM; 12Mo.; Delivered Defect = 0.3 PhaseRqts.Design Code & Unit testIntegration & Test Effort(PM) Cost($K) Schedule(Mo.) Defects in/out/left - Rqts. -Design -Timing -Interface ….. 60/50/10130/116/24264/234/308/37.7/0.3 60/50/1010/16/42/5/11/2/0 120/100/2010/25/52/6.9/0.1 12/6/62/4/41/4.9/0.1 30/25/55/9/10/1/0 ….. 12.543.519 250870380 3.15.83.1 Example of Desired Model - II

14. © USC-CSE 2001 Oct. 24. 200114 Proposed COQUALMO-ODC Approach Simplified initial model Applicability across project types Staged approach Data collection Alternatives to ODC Breakout Group Issues

15. © USC-CSE 2001 Oct. 24. 200115 Estimate defects introduced, removed by phase - Rqts., Design, Code & Unit Test, Integration & Test Use aggregated defect types - One requirements type - Three Design and Code types - Method : Algorithm/method, Assignment/initialization, checking - Static Relationship : relationship, function/class/object, interface, OO messages - Dynamic Relationship : timing/Serialization Use simple Defect Estimating Relationships - Defects introduced : DER (type, phase) - Defects removed : Yield (type, ART, phase) - ART : Analysis, Reviews, Testing Simplified Initial COQUALMO-ODC Model - I

16. © USC-CSE 2001 Oct. 24. 200116 For each defect type (1 Rqts, 3 Design, 3 Code) For each Phase (Rqts, Design, Code & Unit test, Integration & Test) 1. Estimate Defect Density Introduced (DDI) RDDI(t,p) = Reference Defect Density Introduced (from calibration) DM(i) = Defect Multiplier for COCOMO II drivers i Simplified Initial COQUALMO-ODC Model - II

17. © USC-CSE 2001 Oct. 24. 200117 2. Estimate Delivered Defect Density (DDD) - similarly for Code & Unit test, Integration & test Simplified Initial COQUALMO-ODC Model - III

18. © USC-CSE 2001 Oct. 24. 200118 Can be done by < 15 well-qualified people Business rules critical EUI critical Dependability critical Performance Critical Business rules, architecture generally understand COTS-driven Adaptability critical Stakeholder value-driven Method Sweet Spots RAD, XP variants Open source Business-rules driven Specification driven Method generator (MBASE/CeBASE) ODC 1. Small EUI Apps.. 2. Large EUI Apps. 3. Infrastructure 4. Device- Embedded 5. Complex Systems of Systems Critical Discriminators X X X X X X X X X ? X X X X X X X (X) X X X X X X X X X X X X X X X X X X X X X X X X X X X EUI : everyperson user interface (vs. programmer). (X) : most of the time. Business rules : application domain ontologies Project Patterns Software Project Types

19. © USC-CSE 2001 Oct. 24. 200119 Begin with simplified COQUALMO-ODC model - Existing simple requirements defect model - Aggregated categories for design and code defects Begin with Infrastructure applications - where most of data and experience is Experiment with small initial data set - About 5 Infrastructure data points - Plus about 5 USC small-EUI data points Refine and extend based on results, further data Staged COQUALMO-ODC approach

20. © USC-CSE 2001 Oct. 24. 200120 Use currently-collected ODC types - Aggregate later Try for multiple sources - IBM, USC - Exploring Motorola, HP, Telcordia Need COCOMO II cost driver rating for projects - Included on forms - Defined in Book Project Data Collection & Further Research