1. © USC-CSE Feb 6. 20011 Keun Lee ( keunlee@sunset.usc.edu)keunlee@sunset.usc.edu & Sunita Chulani (Sunita_Chulani@us.ibm.com) COQUALMO and Orthogonal Defect Classification(ODC)

2. © USC-CSE Feb 6. 20012 Current COQUALMO Model - Results and Challenges COQUALMO – ODC Research Approach Example Results Issues and Research Plans COQUALMO and Orthogonal Defect Classification(ODC)

3. © USC-CSE Feb 6. 20013 COCOMO II Current COQUALMO System COQUALMO Defect Introduction Model Defect Removal Model Software platform, Project, product and personnel attributes Software Size Estimate Defect removal profile levels Automation, Reviews, Testing Software development effort, cost and schedule estimate Number of residual defects Defect density per unit of size

4. © USC-CSE Feb 6. 20014 Partion of COQUALMO Rating Scale Very LowLowNominalHighVery HighExtra High Automated Analysis Simple compiler syntax checking Basic compiler capabilities Compiler extension Basic req. and design consistency Intermediate- level module Simple req./design More elaborate req./design Basic dist- processing Formalized specification, verification. Advanced dist- processing Peer Reviews No peer review Ad-hoc informal walk-through Well-defined preparation, review, minimal follow-up Formal review roles and Well-trained people and basic checklist Root cause analysis, formal follow Using historical data Extensive review checklist Statistical control Execution Testing and Tools No testingAd-hoc test and debug Basic test Test criteria based on checklist Well-defined test seq. and basic test coverage tool system More advance test tools, preparation. Dist- monitoring Highly advanced tools, model- based test COCOMO II p.263

5. © USC-CSE Feb 6. 20015 COQUALMO Defect Removal Estimates - Nominal Defect Introduction Rates Delivered Defects / KSLOC Composite Defect Removal Rating

6. © USC-CSE Feb 6. 20016 Multiplicative Defect Removal Model - Example : Code Defects; High Ratings Analysis : 0.7 of defects remaining Reviews : 0.4 of defects remaining Testing : 0.31 of defects remaining Together : (0.7)(0.4)(0.31) = 0.09 of defects remaining How valid is this? - All catch same defects : 0.31 of defects remaining - Mostly catch different defects : ~0.01 of defects remaining

7. © USC-CSE Feb 6. 20017 Example UMD-USC CeBASE Data Comparisons “Under specified conditions, …” Peer reviews are more effective than functional testing for faults of omission and incorrect specification(UMD, USC) Functional testing is more effective than reviews for faults concerning numerical approximations and control flow(UMD,USC) Both are about equally effective for results concerning typos, algorithms, and incorrect logic(UMD,USC)

8. © USC-CSE Feb 6. 20018 ODC Data Attractive for Extending COQUALMO - IBM Results (Chillarege, 1996)

9. © USC-CSE Feb 6. 20019 COQUALMO Extension Research Approach Extend COQUALMO to cover major ODC categories Collaborate with industry ODC users - IBM, Motorola underway - Two more sources being explored Obtain first-land experience on USC digital library projects - Completed IBM ODC training - Initial front-end data collection and analysis

10. © USC-CSE Feb 6. 200110 Artifacts - Operational Concept, Requirements, Software Architecture documents Activities - Perspective-based Fagan inspections Triggers - Environment or condition that causes defect Digital Library Analysis to Date - in ODC terms

11. © USC-CSE Feb 6. 200111 Front End (Information Development) Triggers 1.Clarity – confusing or difficulty to understand information. 2.Style – inappropriate or difficulty to understand the manner of expression 3.Accuracy – incorrect information 4.Task Orientation - inappropriate presentation to perform task 5.Organization – relationship between parts is not conveyed 6.Completeness – missing information. 7.Consistency – the expression manner is not displayed in a consist manner

12. © USC-CSE Feb 6. 200112 Initial Digital Library Project ODC Analysis - Trigger percentage Distribution by Team

13. © USC-CSE Feb 6. 200113 Initial Digital Library Project ODC Analysis - Number of Triggers Defects by Team

14. © USC-CSE Feb 6. 200114 Understand anomalies in Digital Library Data - Number of Team 22 defects - Team 4 completeness defects - Due to differences in artifacts or procedures? Continue Digital Library ODC collection & analysis - Detailed Design, code, test Obtain, analyze industry ODC data - Looking for more sources of ODC Data Issues and Research Plans