© USC-CSE 2001 Oct Constructive Quality Model – Orthogonal Defect Classification (COQUALMO-ODC) Model Keun Lee ( & Sunita Chulani

© USC-CSE 2001 Oct COQUALMO, ODC - Current shortfalls COQUALMO-ODC Model Objectives - Example of desired model Breakout Group Issues Outline

© USC-CSE 2001 Oct COCOMO II Current COQUALMO System COQUALMO Defect Introduction Model Defect Removal Model Software platform, Project, product and personnel attributes Software Size Estimate Defect removal profile levels Automation, Reviews, Testing Software development effort, cost and schedule estimate Number of residual defects Defect density per unit of size Defects Introduced Rqts. Design Code

© USC-CSE 2001 Oct Nominal Defect Introduction Rates Delivered Defects / KSLOC Composite Defect Removal Rating COQUALMO Defect Removal Estimates

© USC-CSE 2001 Oct Example : Code Defects; High Ratings Analysis : 0.7 of defects remaining Reviews : 0.4 of defects remaining Testing : 0.31 of defects remaining Together : (0.7)(0.4)(0.31) = 0.09 of defects remaining How valid is this? - All catch same defects : 0.31 of defects remaining - Mostly catch different defects : ~0.01 of defects remaining Need to determine which techniques find which defect type - Orthogonal Defect Classification (ODC) data can help Multiplicative Defect Removal Model

© USC-CSE 2001 Oct ODC A technology for software process measurement and analysis, providing a standard of unique, non-redundant attribute definitions for defect classification as well as analysis metrics and procedures for process evaluations Orthogonal Defect Classification (ODC)

© USC-CSE 2001 Oct Activity/Trigger How defect detected Impact Customer view Interaction Target/Type What was fixed Source Where defect located Qualifier Missing, Incorrect,.. Age History Feedback on Verification ProcessFeedback on Development Process Orthogonal Defect Classification (ODC)

© USC-CSE 2001 Oct IBM Results (Chillarege, 1996) ODC Data Attractive for Extending COQUALMO

© USC-CSE 2001 Oct One-size-fits-all model - May not fit COTS/Web, embedded applications Defect uniformity, independence assumptions - Unvalidated hypotheses COCOMO II C-S-Q trade offs just to RELY levels - Not to delivered defect density, MTBF Need for more calibration data - ODC data could extend and strengthen model Current COQUALMO shortfalls

© USC-CSE 2001 Oct Outline Current COQUALMO Approach - Current shortfalls COQUALMO-ODC Model Objectives - Example of desired model Breakout Group Issues

© USC-CSE 2001 Oct Support cost-schedule-quality tradeoff analysis Provide reference for defect monitoring and control Evolve to cover all major classes of project - With different defect distributions (e.g. COTS-based) - Start simple;grow opportunistically COQUALMO-ODC Model Objectives

© USC-CSE 2001 Oct K 1200K 1800K $20K/PM Effort (PM) VL L N H VH Current: RELY rating Desired: Defect MTBF KSLOC (hr) , Details for any given point- next Time (Mo.) Example of Desired Model - I

© USC-CSE 2001 Oct KSLOC; RELY = VH; 75PM; 12Mo.; Delivered Defect = 0.3 PhaseRqts.Design Code & Unit testIntegration & Test Effort(PM) Cost($K) Schedule(Mo.) Defects in/out/left - Rqts. -Design -Timing -Interface ….. 60/50/10130/116/24264/234/308/37.7/0.3 60/50/1010/16/42/5/11/2/0 120/100/2010/25/52/6.9/0.1 12/6/62/4/41/4.9/0.1 30/25/55/9/10/1/0 … Example of Desired Model - II

© USC-CSE 2001 Oct Proposed COQUALMO-ODC Approach Simplified initial model Applicability across project types Staged approach Data collection Alternatives to ODC Breakout Group Issues

© USC-CSE 2001 Oct Estimate defects introduced, removed by phase - Rqts., Design, Code & Unit Test, Integration & Test Use aggregated defect types - One requirements type - Three Design and Code types - Method : Algorithm/method, Assignment/initialization, checking - Static Relationship : relationship, function/class/object, interface, OO messages - Dynamic Relationship : timing/Serialization Use simple Defect Estimating Relationships - Defects introduced : DER (type, phase) - Defects removed : Yield (type, ART, phase) - ART : Analysis, Reviews, Testing Simplified Initial COQUALMO-ODC Model - I

© USC-CSE 2001 Oct For each defect type (1 Rqts, 3 Design, 3 Code) For each Phase (Rqts, Design, Code & Unit test, Integration & Test) 1. Estimate Defect Density Introduced (DDI) RDDI(t,p) = Reference Defect Density Introduced (from calibration) DM(i) = Defect Multiplier for COCOMO II drivers i Simplified Initial COQUALMO-ODC Model - II

© USC-CSE 2001 Oct Estimate Delivered Defect Density (DDD) - similarly for Code & Unit test, Integration & test Simplified Initial COQUALMO-ODC Model - III

© USC-CSE 2001 Oct Can be done by < 15 well-qualified people Business rules critical EUI critical Dependability critical Performance Critical Business rules, architecture generally understand COTS-driven Adaptability critical Stakeholder value-driven Method Sweet Spots RAD, XP variants Open source Business-rules driven Specification driven Method generator (MBASE/CeBASE) ODC 1. Small EUI Apps.. 2. Large EUI Apps. 3. Infrastructure 4. Device- Embedded 5. Complex Systems of Systems Critical Discriminators X X X X X X X X X ? X X X X X X X (X) X X X X X X X X X X X X X X X X X X X X X X X X X X X EUI : everyperson user interface (vs. programmer). (X) : most of the time. Business rules : application domain ontologies Project Patterns Software Project Types

© USC-CSE 2001 Oct Begin with simplified COQUALMO-ODC model - Existing simple requirements defect model - Aggregated categories for design and code defects Begin with Infrastructure applications - where most of data and experience is Experiment with small initial data set - About 5 Infrastructure data points - Plus about 5 USC small-EUI data points Refine and extend based on results, further data Staged COQUALMO-ODC approach

© USC-CSE 2001 Oct Use currently-collected ODC types - Aggregate later Try for multiple sources - IBM, USC - Exploring Motorola, HP, Telcordia Need COCOMO II cost driver rating for projects - Included on forms - Defined in Book Project Data Collection & Further Research