1. SOFTWARE QUALITY IN 2000: WHAT WORKS AND WHAT DOESN’T Copyright © 2000 by SPR. All Rights Reserved. Software Productivity Research an Artemis company One New England Executive Park Burlington, Massachusetts 01803 781.273.0140 Fax: 781.273.5176 http://www.spr.com January 5, 2000 Capers Jones, Chief Scientist

2. SWQUAL97\2 Copyright © 2000 by SPR. All Rights Reserved. LESSONS FOR SOFTWARE QUALITY IN 2000 LESSONS FROM THE Y2K PROBLEM Businesses are tightly coupled in “supply chains.” Poor quality in one company can affect scores of companies. The press only covers immediate problems - not long range. Government software affects all businesses. Quality and security are becoming intertwined. Reuse and components depend upon high quality. Web-based applications need both quality and security.

3. SWQUAL97\3 Copyright © 2000 by SPR. All Rights Reserved. BASIC DEFINITIONS SOFTWARESoftware that combines the QUALITYcharacteristics of low defect rates and high user satisfaction USERClients who are pleased with a SATISFACTIONvendor’s products, quality levels, ease of use, and support DEFECTTechnologies that minimize the PREVENTIONrisk of making errors in software deliverables DEFECTActivities that find and correct REMOVALdefects in software deliverables BAD FIXESSecondary defects injected as a byproduct of defect repairs

4. SWQUAL97\4 Copyright © 2000 by SPR. All Rights Reserved. FUNDAMENTAL SOFTWARE QUALITY METRICS Defect Potentials –requirements errors, design errors, code errors, document errors, bad fix errors, test plan errors, and test case errors Defects Removed –by origin of defects –before testing –during testing –during deployment Defect Removal Efficiency –ratio of development defects to customer defects Defect Severity Levels (Valid defects) –fatal, serious, minor, cosmetic

5. SWQUAL97\5 Copyright © 2000 by SPR. All Rights Reserved. Duplicate Defects Invalid Defects Defect Removal Effort and Costs –preparation –execution –repairs and rework –effort on duplicates and invalids Supplemental Quality Metrics –complexity –test case volumes –test case coverage –IBM’s orthogonal defect categories FUNDAMENTAL SOFTWARE QUALITY METRICS (cont.)

6. SWQUAL97\6 Copyright © 2000 by SPR. All Rights Reserved. Standard Cost of Quality –Prevention –Appraisal –Failures Error-Prone Module Effort –Identification –Removal or redevelopment –repairs and rework Mass-Update Effort (Year 2000, Euro, etc.) FUNDAMENTAL SOFTWARE QUALITY METRICS (cont.)

7. SWQUAL97\7 Copyright © 2000 by SPR. All Rights Reserved. HAZARDOUS QUALITY DEFINITIONS Quality means conformance to requirements Requirements contain > 15% of software errors. Requirements grow at > 2% per month. Do you conform to requirements errors? Do you conform to totally new requirements? Whose requirements are you trying to satisfy?

8. SWQUAL97\8 Copyright © 2000 by SPR. All Rights Reserved. HAZARDOUS QUALITY METRICS Cost per Defect Approaches infinity as defects near zero Conceals real economic value of quality

9. SWQUAL97\9 Copyright © 2000 by SPR. All Rights Reserved. COST PER DEFECT PENALIZES QUALITY ABCD PoorGoodExcellentZero QualityQualityQualityDefects Function Points100100100100 Bugs Discovered5005050 Preparation$5,000$5,000$5,000$5,000 Removal$5,000$2,500$1,000$ 0 Repairs$25,000$5,000$1,000$ 0 Total$35,000$12,500$7,000$5,000 Cost per Defect$70$250$1,400 Cost per Function Point$350$125$70$50

10. SWQUAL97\10 Copyright © 2000 by SPR. All Rights Reserved. HAZARDS OF “DEFECTS PER KLOC” METRICS Defects per KLOC When defects are found in multiple deliverables, it is invalid to assign all defects to a single item. Software defects are found in: requirements design source code user documents bad fixes (secondary defects) Requirements and design defects outnumber code defects. Defects per KLOC metrics makes major sources of software defects invisible and constitutes professional malpractice.

11. SWQUAL97\11 Copyright © 2000 by SPR. All Rights Reserved. FOUR LANGUAGE COMPARISON OF SOFTWARE DEFECT POTENTIALS Defect OriginAssemblyAda C ++ C++ and Reuse Function points 100100 100 100 KLOC 30 7.5 5.5 2.5 Requirements20202020 Design50503515 Code150453515 Documents25252525 Bad Fixes20107 4 TOTAL DEFECTS 265150 12279 Defects per KLOC10.620.0 22.2 31.6 Defects/Function Point3.0 2.0 1.22 0.79 Defect per KLOC may be considered to be professional malpractice.

12. SWQUAL97\12 Copyright © 2000 by SPR. All Rights Reserved. Assembly C Fortran COBOL PL/I ADA83 ADA9X C++ Generators Smalltalk LOC VERSUS FUNCTION POINT QUALITY LEVELS Defects per KLOC

13. SWQUAL97\13 Copyright © 2000 by SPR. All Rights Reserved. GRAPH OF MAJOR SOFTWARE RISKS Schedule Slippage Unplanned Changes Inadequate Defect Removal High Defect Levels Lawsuit for Missed Schedule Poor User Satisfaction Deferred Functions Cost Overrun

14. SWQUAL97\14 Copyright © 2000 by SPR. All Rights Reserved. Defect RemovalDelivered Defect OriginsPotentialEfficiency Defects Requirements1.0077%0.23 Design1.2585%0.19 Coding1.7595%0.09 Documents0.6080%0.12 Bad Fixes0.4070%0.12 TOTAL5.0085%0.75 (Data expressed in terms of defects per function point) U.S. AVERAGES FOR SOFTWARE QUALITY (Function points show all defect sources - not just coding defects)

15. SWQUAL97\15 Copyright © 2000 by SPR. All Rights Reserved. OBSERVATIONS Most often found in military and systems software > SEI CMM Level 3 Defect RemovalDelivered Defect OriginsPotentialEfficiency Defects Requirements0.4085%0.08 Design0.6097%0.02 Coding1.0099%0.01 Documents0.4098%0.01 Bad Fixes0.1095%0.01 TOTAL2.5096%0.13 (Data expressed in terms of defects per function point) BEST IN CLASS SOFTWARE QUALITY

16. SWQUAL97\16 Copyright © 2000 by SPR. All Rights Reserved. OBSERVATIONS Most often found in large client-server projects (> 5000 FP). Defect RemovalDelivered Defect OriginsPotentialEfficiency Defects Requirements1.5050%0.75 Design2.2050%1.10 Coding2.5080%0.50 Documents1.0070%0.30 Bad Fixes0.8050%0.40 TOTAL8.0062%3.05 (Data expressed in terms of defects per function point) POOR SOFTWARE QUALITY - MALPRACTICE

17. SWQUAL97\17 Copyright © 2000 by SPR. All Rights Reserved. INTERNATIONAL QUALITY COMPARISONS CountryDefectRemovalDelivered PotentialEfficiencyDefects Japan4.5093%0.32 Canada4.5586%0.64 United States5.0085%0.75 Norway4.9584%0.79 Sweden5.0084%0.80 France4.7583%0.82 Italy4.8583%0.82 India5.1084%0.82 Germany4.9583%0.84 England4.8582%0.87 South Korea5.2083%0.88 Russia5.5080%1.10

18. SWQUAL97\18 Copyright © 2000 by SPR. All Rights Reserved. Formal Inspections (Design and Code) Joint Application Design (JAD) Quality Function Deployment (QFD) Quality Metrics using function points Quality Metrics using IBM’s Orthogonal classification Defect Removal Efficiency Measurements Defect tracking tools Active Quality Assurance (> 5% QA staff) Formal Configuration Control User Satisfaction Surveys Formal Test Planning Quality Estimation Tools Automated Test Tools Testing Specialists Root-Cause Analysis GOOD QUALITY RESULTS > 90% OF COMPANIES

19. SWQUAL97\19 Copyright © 2000 by SPR. All Rights Reserved. MIXED QUALITY RESULTS: < 50% SUCCESS Total Quality Management (TQM) Independent Verification & Validation (IV & V) Independent quality audits SEI Assessments and the CMM Six-Sigma quality programs Baldrige Awards IEEE Quality Standards Testing only by Developers DOD 2167A and DOD 498 Reliability Models Quality circles Clean-room methods Cost of quality

20. SWQUAL97\20 Copyright © 2000 by SPR. All Rights Reserved. MINIMAL QUALITY RESULTS: < 25% SUCCESS ISO 9000 - 9004 Quality Standards Informal Testing Manual Testing Passive Quality Assurance (< 3% QA staff) Token Quality Assurance (< 1% QA staff) LOC Metrics Cost per defect metric Rapid Application Development (RAD)

21. SWQUAL97\21 Copyright © 2000 by SPR. All Rights Reserved. QUALITY METHOD EFFECTIVENESS AND COSTS METHOD EFFECTIVENESS COSTS Formal InspectionsVery HighHigh Defect EstimationVery HighLow Defect TrackingHighLow Formal TestingHighHigh QA OrganizationHighHigh Independent auditsHighHigh JAD and QFDHighLow PrototypingHighLow Test Case ToolsHighMedium Change TrackingHighMedium Informal WalkthroughsModerateMedium Informal TestingModerateMedium TQMModerateMedium ISO 9000-9004MarginalHigh

22. SWQUAL97\22 Copyright © 2000 by SPR. All Rights Reserved. A PRACTICAL DEFINITION OF SOFTWARE QUALITY (PREDICTABLE AND MEASURABLE) Low Defect Potentials (< 2.5 per Function Point) High Defect Removal Efficiency (> 95%) Unambiguous, Stable Requirements (< 2.5% change) Explicit Requirements Achieved (> 97.5% achieved) High User Satisfaction Ratings (> 90% “excellent”) - Installation - Ease of learning - Ease of use - Functionality - Compatibility - Error handling - User information (screens, manuals, tutorials) - Customer support - Defect repairs

23. SWQUAL97\23 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE QUALITY OBSERVATIONS Individual programmers -- Less than 50% efficient in finding bugs in their own software Normal test steps -- often less than 70% efficient (1 of 3 bugs remain) Design Reviews and Code Inspections -- often more than 65% efficient; have topped 85% Reviews or inspections plus formal testing -- are often more than 96% efficient; have hit 99% Reviews and Inspections -- lower costs and schedules by as much as 30 % Quality Measurements Have Found:

24. SWQUAL97\24 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT ORIGINS 1)Requirements: Hardest to prevent and repair 2)Design: Most severe and pervasive 3)Code: Most numerous; easiest to fix 4)Documentation: Can be serious if ignored 5)Bad Fixes: Very difficult to find 6)Bad Test Cases: Common and troublesome 7)Data quality: Common and unmeasurable

25. SWQUAL97\25 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT SEVERITIES Severity 1:TOTAL FAILURES 1% at release Severity 2:MAJOR PROBLEMS20% at release Severity 3:MINOR PROBLEMS35% at release Severity 4:COSMETIC ERRORS44% at release INVALIDUSER OR SYSTEM ERRORS15% of reports DUPLICATEMULTIPLE REPORTS30% of reports ABEYANTCAN’T RECREATE ERROR 5% of reports

26. SWQUAL97\26 Copyright © 2000 by SPR. All Rights Reserved. WORK CATEGORIES RELATED TO PRODUCT SIZE 1.6 1.4 1.2 1.0.8.6.4.2 Size of Program in Function Points 1020408016032064012802560512010240 Defect Related Project Related Code Related

27. SWQUAL97\27 Copyright © 2000 by SPR. All Rights Reserved. PERCENTAGE OF SOFTWARE EFFORT BY TASK Size inMgt./Defect Function PointsSupportRemovalPaperworkCodingTotal 10,24018%36%34%12%100% 5,12017%33%32%18%100% 2,58016%31%29%24%100% 1,28015%29%26%30%100% 64014%27%23%36%100% 32013%25%20%42%100% 16012%23%17%48%100% 8011%21%14%54%100% 4010%19%11%60%100% 209%17%8%66%100% 108%15%5%72%100%

28. SWQUAL97\28 Copyright © 2000 by SPR. All Rights Reserved. HOW QUALITY AFFECTS SOFTWARE COSTS COST TIME Pathological Healthy Poor quality is cheaper until the end of the coding phase. After that, high quality is cheaper.

29. SWQUAL97\29 Copyright © 2000 by SPR. All Rights Reserved. U. S. SOFTWARE QUALITY AVERAGES (Defects per Function Point)) SystemCommercialInformationMilitaryOverall SoftwareSoftwareSoftwareSoftwareAverage Defect Potentials6.05.04.57.05.6 Defect Removal94%90%73%96%88% Efficiency Delivered Defects0.40.51.20.30.65 First Year Discovery Rate65%70%30%75%60% First Year Reported0.260.350.360.230.30 Defects

30. SWQUAL97\30 Copyright © 2000 by SPR. All Rights Reserved. RELATIONSHIP BETWEEN SOFTWARE SIZE AND DEFECT REMOVAL EFFICIENCY (Data Expressed in terms of Defects per Function Point)

31. SWQUAL97\31 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT POTENTIALS AND DEFECT REMOVAL EFFICIENCY FOR EACH LEVEL OF SEI CMM (Data Expressed in Terms of Defects per Function Point) DefectRemovalDelivered SEI CMM LevelsPotentialsEfficiencyDefects SEI CMM 15.0085%0.75 SEI CMM 24.0090%0.40 SEI CMM 33.0095%0.15 SEI CMM 42.0097%0.08 SEI CMM 51.0099%0.01

32. SWQUAL97\32 Copyright © 2000 by SPR. All Rights Reserved. MAJOR SOFTWARE QUALITY ZONES 10 9 8 7 6 5 4 3 2 1 0 50%55%60%65%70%75%80%85%90%95%100% Defect Removal Efficiency Malpractice U.S. and Europe Average Best In Class Canada India Japan

33. SWQUAL97\33 Copyright © 2000 by SPR. All Rights Reserved. Defect Removal Efficiency Defects per FP Malpractice U.S. Average Best in Class.   MAJOR SOFTWARE QUALITY ZONES SEI CMM 2 SEI CMM 3 SEI CMM 4 SEI CMM 5 The SEI CMM has overlaps among the levels.

34. SWQUAL97\34 Copyright © 2000 by SPR. All Rights Reserved. Defect Removal Efficiency Defects per FP Malpractice U.S. Average Best in Class.    Client/Server MAJOR SOFTWARE QUALITY ZONES Client-server projects are worse than U.S. averages

35. SWQUAL97\35 Copyright © 2000 by SPR. All Rights Reserved. Defect Removal Efficiency Defects per FP Malpractice U.S. Average Best in Class.    Telecommunications SOFTWARE QUALITY IMPROVEMENT (cont.) Telecommunications projects are better than U.S. averages

36. SWQUAL97\36 Copyright © 2000 by SPR. All Rights Reserved. Defect Removal Efficiency Defects per FP Malpractice U.S. Average Best in Class.   Object-oriented  SOFTWARE QUALITY IMPROVEMENT (cont.) OO projects can be hazardous due to steep learning curve

37. SWQUAL97\37 Copyright © 2000 by SPR. All Rights Reserved. Defect Removal Efficiency Defects per FP Malpractice U.S. Average Best in Class.    ISO 9001 SOFTWARE QUALITY IMPROVEMENT (cont.) ISO 9000-9004 have uncertain results

38. SWQUAL97\38 Copyright © 2000 by SPR. All Rights Reserved. Defect Removal Efficiency Defects per FP Malpractice U.S. Average Best in Class.   SOFTWARE QUALITY IMPROVEMENT (cont.)  DoD 2167A DoD 498 Military projects are better than U.S. averages

39. SWQUAL97\39 Copyright © 2000 by SPR. All Rights Reserved. INDUSTRY-WIDE DEFECT CAUSES 1. Requirements problems (omissions; changes) 2.Design problems 3.Interface problems between modules 4.Logic, branching, and structural problems 5.Memory allocation problems 6.Testing omissions and poor coverage 7. Test case errors 8.Stress/performance problems 9.Bad fixes/Regressions 10.Documentation errors Ranked in order of effort required to fix the defects:

40. SWQUAL97\40 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE QUALITY UNKNOWNS ERRORS IN SOFTWARE TEST PLANS ERRORS IN SOFTWARE TEST CASES MASS-UPDATE TESTING (YEAR 2000 AND EURO) SUPPLY-CHAIN TESTING (MULTI-NATIONAL) ERRORS IN DATA BASES AND DATA WAREHOUSES CAUSES OF BAD FIX INJECTION RATES IMPACT OF COMPLEXITY ON QUALITY IMPACT OF CREEPING REQUIREMENTS SOFTWARE QUALITY TOPICS NEEDING RESEARCH:

41. SWQUAL97\41 Copyright © 2000 by SPR. All Rights Reserved. U.S. INDUSTRIES EXCEEDING 95% IN CUMULATIVE DEFECT REMOVAL EFFICIENCY Year 95% Exceeded (Approximate) 1.Telecommunications Manufacturing1975 2.Computer Manufacturing1977 3.Aero-space Manufacturing 1979 4.Military and Defense Manufacturing 1980 5.Medical Instrument Manufacturing1980 6.Commercial Software Producers 1992

42. SWQUAL97\42 Copyright © 2000 by SPR. All Rights Reserved. U.S. INDUSTRIES MAINTAINING MARKET SHARE INTERNATIONALLY 1.Telecommunications Manufacturing 2.Computer Manufacturing 3.Military and Defense Manufacturing 4.Commercial Software Producers 5.Aero-space Manufacturing 6.Medical Instrument Manufacturing

43. SWQUAL97\43 Copyright © 2000 by SPR. All Rights Reserved. DEFECT REMOVAL AND TESTING STAGES NOTED DURING LITIGATION FOR POOR QUALITY

44. SWQUAL97\44 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE QUALITY AND LITIGATION PLAINTIFF CLAIMS: DEFENDANT CLAIMS: Schedule overrun Requirements changes Cost overrun New demands by clients Poor quality Rushed by clients False claims Refusal to cooperate PROBLEMS ON BOTH SIDES Ambiguous clauses in contract Informal software cost estimates No formal quality estimates at all No use of formal inspections Inadequate milestone tracking Friction and severe personal disputes Independent audits too late to solve issues

45. SWQUAL97\45 Copyright © 2000 by SPR. All Rights Reserved. INDEPENDENT ASSESSMENTS AND AUDITS Often used for military projects Starting to be used for Year 2000 and Euro updates Can be an effective defense for litigation Effective quality assessments are formal Effective quality assessments cover defect prevention Effective quality assessments cover defect removal Effective quality assessments cover defect measures Effective assessments should cover 100% of projects Samples or partial assessments not safe for litigation

46. SWQUAL97\46 Copyright © 2000 by SPR. All Rights Reserved. OPTIMIZING QUALITY AND PRODUCTIVITY Projects that achieve 95% cumulative Defect Removal Efficiency will find: 1) Minimum schedules 2) Maximum productivity 3) High levels of user satisfaction 4) Low levels of delivered defects 5) Low levels of maintenance costs

47. SWQUAL97\47 Copyright © 2000 by SPR. All Rights Reserved. DEFECT REMOVAL AND SCHEDULE DELAYS Project Size Expressed in Function Points Project Outcome100100010,000100,000 Late Projects Defect potentials3.004.255.006.50 Defect removal efficiency83%74%67%63% Delivered defects0.511.111.652.40 On-Time Projects Defect potentials2.804.004.706.00 Defect removal efficiency96%92%90%87% Delivered defects0.110.320.470.78 Early Projects Defect potentials2.603.854.505.50 Defect removal efficiency98%96%94%92% Delivered defects0.050.150.270.44

48. SWQUAL97\48 Copyright © 2000 by SPR. All Rights Reserved. ORIGINS OF SOFTWARE DEFECTS Because defect removal is such a major cost element, studying defect origins is a valuable undertaking. IBM Corporation (MVS)SPR Corporation (client studies) 45%Design errors20%Requirements errors 25%Coding errors30%Design errors 20%Bad fixes35%Coding errors 5%Documentation errors10%Bad fixes 5%Administrative errors 5%Documentation errors100% TRW CorporationMitre CorporationNippon Electric Corp. 60%Design errors64%Design errors60%Design errors 40%Coding errors 36%Coding errors 40%Coding errors 100%100%100%

49. SWQUAL97\49 Copyright © 2000 by SPR. All Rights Reserved. FUNCTION POINTS AND DEFECT POTENTIALS Function points raised to the 1.15 power can predict the probable number of software defects. The range is from 1.1 to 1.25 power. (Defects in requirements, design, code, documents, and bad fix categories.) FUNCTION POINTSPOTENTIAL DEFECTS 1 1014 100200 1,0002,818 10,00039,811 100,000316,228

50. SWQUAL97\50 Copyright © 2000 by SPR. All Rights Reserved. RELATIONSHIP OF SOFTWARE QUALITY AND PRODUCTIVITY The most effective way of improving software productivity and shortening project schedules is to reduce defect levels. Defect reduction can occur through: 1.Defect prevention technologies Structured design Structured code High-level languages Etc. 2.Defect removal technologies Design reviews Code inspections Tests Correctness proofs

51. SWQUAL97\51 Copyright © 2000 by SPR. All Rights Reserved. DEFECT PREVENTION AND DEFECT REMOVAL DEFECT PREVENTION Joint Application Design (JAD) Quality function deployment (QFD) Software reuse (high-quality) Root cause analysis Six-Sigma quality programs ISO 9000-9004 audits

52. SWQUAL97\52 Copyright © 2000 by SPR. All Rights Reserved. DEFECT PREVENTION AND DEFECT REMOVAL DEFECT PREVENTION SEI CMM assessments SPR assessments SPR benchmarks TickIT assessments SPICE assessments Kaizen methodology Quality circles Independent Verification & Validation (IV&V)

53. SWQUAL97\53 Copyright © 2000 by SPR. All Rights Reserved. DEFECT PREVENTION AND DEFECT REMOVAL DEFECT PREVENTION Total quality management (TQM) Quality measurements Orthogonal defect analysis Defect tracking tools Formal design inspections Formal code inspections

54. SWQUAL97\54 Copyright © 2000 by SPR. All Rights Reserved. DEFECT PREVENTION AND DEFECT REMOVAL DEFECT REMOVAL Requirements inspections Design inspections Test plan inspections Test case inspections Code inspections User manual inspections Data quality inspections

55. SWQUAL97\55 Copyright © 2000 by SPR. All Rights Reserved. DEFECT PREVENTION AND DEFECT REMOVAL DEFECT REMOVAL Independent audits Testing: normal forms Testing: special forms Testing: user-based forms Testing: independent Testing: clean-room

56. SWQUAL97\56 Copyright © 2000 by SPR. All Rights Reserved. DEFECT PREVENTION METHODS JAD’s Prototypes Structured Methods Design Tools Blueprints & Reusable Code QFD RequirementsDesignCodeDocumentPerformance DefectsDefectsDefectsDefectsDefects

57. SWQUAL97\57 Copyright © 2000 by SPR. All Rights Reserved. DEFECT REMOVAL METHODS Reviews/ Inspections Prototypes Testing (all forms) Correctness Proofs RequirementsDesignCodeDocumentPerformance DefectsDefectsDefectsDefectsDefects FairExcellentExcellentGoodFair Not GoodFairFairApplicableGood PoorPoorGoodFairExcellent PoorPoorGoodFairPoor

58. SWQUAL97\58 Copyright © 2000 by SPR. All Rights Reserved. QUALITY MEASUREMENT EXCELLENCE Test DefectDefectUsabilityComplexityCoverageRemovalMaintenance EstimationTrackingMeasuresMeasuresMeasuresMeasuresMeasures 1. ExcellentYesYesYesYesYesYesYes 2.GoodYesYesYesNoYesNoYes 3.AverageNoYesYesNoYesNoYes 4.MarginalNoNoYesNoYesNoYes 5.PoorNoNoNoNoNoNoNo

59. SWQUAL97\59 Copyright © 2000 by SPR. All Rights Reserved. DEFECT REMOVAL EFFICIENCY Removal efficiency is the most important quality measure Defects found Removal efficiency = Defects present “Defects present” is the critical parameter

60. SWQUAL97\60 Copyright © 2000 by SPR. All Rights Reserved. DEFECT REMOVAL EFFICIENCY (cont.) 12345678910 Defects First operation 6 defects from 10 or 60% efficiency Second operation 2 defects from 4 or 50% efficiency Cumulative efficiency 8 defects from 10 or 80% efficiency Defect removal efficiency =Percentage of defects removed by a single level of review, inspection or test Cumulative defect removal efficiency =Percentage of defects removed by a series of reviews, inspections or tests

61. SWQUAL97\61 Copyright © 2000 by SPR. All Rights Reserved. DEFECT REMOVAL EFFICIENCY EXAMPLE DEVELOPMENT DEFECTS Inspections500 Testing400 Subtotal900 USER-REPORTED DEFECTS IN FIRST 90 DAYS Valid unique100 TOTAL DEFECT VOLUME Defect totals 1000 REMOVAL EFFICIENCY Dev. (900) / Total (1000) =90%

62. SWQUAL97\62 Copyright © 2000 by SPR. All Rights Reserved. RANGES OF DEFECT REMOVAL EFFICIENCY LowestMedianHighest 1 Requirements review20%30%50% 2 Top-level design reviews30%40%60% 3 Detailed functional design reviews30%45%65% 4 Detailed logic design reviews35%55%75% 5 Code inspections35%60%85% 6 Unit tests10%25%50% 7 New Function tests20%35%55% 8 Integration tests25%45%60% 9 System test25%50%65% 10 External Beta tests15%40%75% CUMULATIVE EFFICIENCY75%97%99.99%

63. SWQUAL97\63 Copyright © 2000 by SPR. All Rights Reserved. NORMAL DEFECT ORIGIN/DISCOVERY GAPS Defect Origins Defect Discovery Requirements DesignCodingDocumentationTestingMaintenance Requirements DesignCodingDocumentationTestingMaintenance Zone of Chaos

64. SWQUAL97\64 Copyright © 2000 by SPR. All Rights Reserved. Defect Origins Defect Discovery Requirements DesignCodingDocumentationTestingMaintenance Requirements DesignCodingDocumentationTestingMaintenance DEFECT ORIGINS/DISCOVERY WITH INSPECTIONS

65. SWQUAL97\65 Copyright © 2000 by SPR. All Rights Reserved. TECHNOLOGY COMBINATIONS DEFECT REMOVAL EFFICIENCY LowestMedianHighest 1.No Design Inspections30%40%50% No Code Inspections No Quality Assurance No Formal Testing WORST CASE RANGE SOFTWARE DEFECT REMOVAL RANGES

66. SWQUAL97\66 Copyright © 2000 by SPR. All Rights Reserved. TECHNOLOGY COMBINATIONS DEFECT REMOVAL EFFICIENCY LowestMedianHighest 2.No design inspections32%45%55% No code inspections FORMAL QUALITY ASSURANCE No formal testing 3.No design inspections37%53%60% No code inspections No quality assurance FORMAL TESTING 4.No design inspections43%57%65% FORMAL CODE INSPECTIONS No quality assurance No formal testing 5.FORMAL DESIGN INSPECTIONS45%60%68% No code inspections No quality assurance No formal testing SOFTWARE DEFECT REMOVAL RANGES (cont.) SINGLE TECHNOLOGY CHANGES

67. SWQUAL97\67 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT REMOVAL RANGES (cont.) TECHNOLOGY COMBINATIONS DEFECT REMOVAL EFFICIENCY LowestMedianHighest 6.No design inspections50%65%75% No code inspections FORMAL QUALITY ASSURANCE FORMAL TESTING 7.No design inspections53%68%78% FORMAL CODE INSPECTIONS FORMAL QUALITY ASSURANCE No formal testing 8.No design inspections55%70%80% FORMAL CODE INSPECTIONS No quality assurance FORMAL TESTING TWO TECHNOLOGY CHANGES

68. SWQUAL97\68 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT REMOVAL RANGES (cont.) TECHNOLOGY COMBINATIONS DEFECT REMOVAL EFFICIENCY LowestMedianHighest 9.FORMAL DESIGN INSPECTIONS60%75%85% No code inspections FORMAL QUALITY ASSURANCE No formal testing 10.FORMAL DESIGN INSPECTIONS65%80%87% No code inspections No quality assurance FORMAL TESTING 11.FORMAL DESIGN INSPECTIONS70%85%90% FORMAL CODE INSPECTIONS No quality assurance No formal testing TWO TECHNOLOGY CHANGES (cont.)

69. SWQUAL97\69 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT REMOVAL RANGES (cont.) TECHNOLOGY COMBINATIONS DEFECT REMOVAL EFFICIENCY LowestMedianHighest 12.No design inspections75%87%93% FORMAL CODE INSPECTIONS FORMAL QUALITY ASSURANCE FORMAL TESTING 13.FORMAL DESIGN INSPECTIONS77%90%95% No code inspections FORMAL QUALITY ASSURANCE FORMAL TESTING 14.FORMAL DESIGN INSPECTIONS83%95%97% FORMAL CODE INSPECTIONS FORMAL QUALITY ASSURANCE No formal testing 15.FORMAL DESIGN INSPECTIONS85%97%99% FORMAL CODE INSPECTIONS No quality assurance FORMAL TESTING THREE TECHNOLOGY CHANGES

70. SWQUAL97\70 Copyright © 2000 by SPR. All Rights Reserved. SOFTWARE DEFECT REMOVAL RANGES (cont.) TECHNOLOGY COMBINATIONS DEFECT REMOVAL EFFICIENCY LowestMedianHighest 1.FORMAL DESIGN INSPECTIONS95%99%99.99% FORMAL CODE INSPECTIONS FORMAL QUALITY ASSURANCE FORMAL TESTING BEST CASE RANGE

71. SWQUAL97\71 Copyright © 2000 by SPR. All Rights Reserved. DISTRIBUTION OF 1500 SOFTWARE PROJECTS BY DEFECT REMOVAL EFFICIENCY LEVEL

72. SWQUAL97\72 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY SOFTWARE QUALITY METHODS VARY BY CLASS: 1) Systems software 2) Embedded software 3) Military software 4) Commercial software 5) Outsourced software 6) Management information systems (MIS) software 7) End - User developed software

73. SWQUAL97\73 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY SYSTEMS SOFTWARE QUALITY METHODS USUALLY > 96% DEFECT REMOVAL EFFICIENCY OVERALL, BEST SOFTWARE QUALITY RESULTS BEST QUALITY RESULTS > 10,000 FUNCTION POINTS FORMAL DESIGN AND CODE INSPECTIONS FORMAL SOFTWARE QUALITY ASSURANCE GROUPS FORMAL SOFTWARE QUALITY MEASUREMENTS FORMAL CHANGE CONTROL FORMAL TEST PLANS UNIT TEST BY DEVELOPERS 6 TO 10 TEST STAGES BY TEST SPECIALISTS

74. SWQUAL97\74 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY EMBEDDED SOFTWARE QUALITY METHODS USUALLY > 90% DEFECT REMOVAL EFFICIENCY MOST PROJECTS < 500 FUNCTION POINTS IN SIZE WIDE RANGE OF SOFTWARE QUALITY RESULTS SHOULD USE FORMAL INSPECTIONS, BUT MAY NOT SHOULD USE FORMAL SQA TEAMS, BUT MAY NOT INFORMAL SOFTWARE QUALITY MEASUREMENTS SHOULD USE FORMAL CHANGE CONTROL SHOULD USE FORMAL TEST PLANS UNIT TEST BY DEVELOPERS 3 TO 6 TEST STAGES SHOULD USE TEST SPECIALISTS, BUT MAY NOT

75. SWQUAL97\75 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY MILITARY SOFTWARE QUALITY METHODS USUALLY > 95% DEFECT REMOVAL EFFICIENCY OVERALL, GOOD SOFTWARE QUALITY RESULTS BEST QUALITY RESULTS > 100,000 FUNCTION POINTS FORMAL DESIGN AND CODE INSPECTIONS FORMAL SOFTWARE QUALITY ASSURANCE GROUPS FORMAL SOFTWARE QUALITY MEASUREMENTS FORMAL CHANGE CONTROL FORMAL TEST PLANS UNIT TEST BY DEVELOPERS 6 TO 15 TEST STAGES BY TEST SPECIALISTS ONLY CLASS TO USE INDEPENDENT VERIF. AND VALID. ONLY CLASS TO USE INDEPENDENT TESTING

76. SWQUAL97\76 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY COMMERCIAL SOFTWARE QUALITY METHODS USUALLY > 90% DEFECT REMOVAL EFFICIENCY MOST PROJECTS > 3000 FUNCTION POINTS IN SIZE WIDE RANGE OF SOFTWARE QUALITY RESULTS SHOULD USE FORMAL INSPECTIONS, BUT MAY NOT SHOULD USE FORMAL SQA TEAMS, BUT MAY NOT INFORMAL SOFTWARE QUALITY MEASUREMENTS FORMAL CHANGE CONTROL FORMAL TEST PLANS UNIT TEST BY DEVELOPERS 3 TO 8 TEST STAGES SHOULD USE TEST SPECIALISTS, BUT MAY NOT OFTEN EXTENSIVE BETA TESTING BY USERS

77. SWQUAL97\77 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY OUTSOURCED SOFTWARE QUALITY METHODS USUALLY > 94% DEFECT REMOVAL EFFICIENCY OVERALL, BETTER SOFTWARE QUALITY THAN CLIENTS GOOD QUALITY > 1000 FUNCTION POINTS SHOULD USE FORMAL INSPECTIONS, BUT MAY NOT SHOULD USE FORMAL SQA GROUPS, BUT MAY NOT SHOULD USE FORMAL QUALITY MEASUREMENTS SHOULD USE FORMAL CHANGE CONTROL SHOULD USE FORMAL TEST PLANS UNIT TEST BY DEVELOPERS 4 TO 8 TEST STAGES BY TEST SPECIALISTS ACCEPTANCE TESTING BY CLIENTS MANY LATE CHANGES DEMANDED BY CLIENTS

78. SWQUAL97\78 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY MIS SOFTWARE QUALITY METHODS USUALLY < 90% DEFECT REMOVAL EFFICIENCY OFTEN MEDIOCRE SOFTWARE QUALITY POOR QUALITY > 1000 FUNCTION POINTS SELDOM USES FORMAL DESIGN AND CODE INSPECTIONS SELDOM USES FORMAL SQA GROUPS SELDOM USES SOFTWARE QUALITY MEASUREMENTS FORMAL CHANGE CONTROL INFORMAL TEST PLANS UNIT TEST BY DEVELOPERS 2 TO 6 TEST STAGES BY DEVELOPERS ACCEPTANCE TESTING BY CLIENTS

79. SWQUAL97\79 Copyright © 2000 by SPR. All Rights Reserved. PATTERNS OF SOFTWARE QUALITY END-USER SOFTWARE QUALITY METHODS USUALLY < 50% DEFECT REMOVAL EFFICIENCY OFTEN DANGEROUSLY POOR SOFTWARE QUALITY ALL PROJECTS < 100 FUNCTION POINTS NO USE OF FORMAL DESIGN AND CODE INSPECTIONS NO USE OF SQA GROUPS NO USE OF SOFTWARE QUALITY MEASUREMENTS INFORMAL CHANGE CONTROL SELDOM ANY TEST PLANS UNIT TEST BY DEVELOPER MAY BE ONLY TEST STAGE

80. SWQUAL97\80 Copyright © 2000 by SPR. All Rights Reserved. AVERAGE NUMBER OF TEST STAGES OBSERVED BY APPLICATION SIZE AND CLASS OF SOFTWARE (Size of Application in Function Points)

81. SWQUAL97\81 Copyright © 2000 by SPR. All Rights Reserved. NUMBER OF TESTING STAGES, TESTING EFFORT, AND DEFECT REMOVAL EFFICIENCY Number ofPercent of EffortCumulative Defect Testing StagesDevoted to TestingRemoval Efficiency 1 testing stage10%50% 2 testing stages15%60% 3 testing stages20%70% 4 testing stages25%75% 5 testing stages30%80% 6 testing stages*33%*85%* 7 testing stages36%87% 8 testing stages39%90% 9 testing stages42%92% *Note: Six test stages, 33% costs, and 85% removal efficiency are U.S. averages.

82. SWQUAL97\82 Copyright © 2000 by SPR. All Rights Reserved. NUMBER OF TESTING STAGES, TESTING EFFORT, AND DEFECT REMOVAL EFFICIENCY (cont.) Number ofPercent of EffortCumulative Defect Testing StagesDevoted to TestingRemoval Efficiency 10 testing stages45%94% 11 testing stages48%96% 12 testing stages52%98% 13 testing stages55%99% 14 testing stages58%99.9% 15 testing stages61%99.99% 16 testing stages64%99.999% 17 testing stages67%99.9999% 18 testing stages70%99.99999% *Note: Six test stages, 33% costs, and 85% removal efficiency are U.S. averages.

83. SWQUAL97\83 Copyright © 2000 by SPR. All Rights Reserved. CONCLUSIONS ON SOFTWARE DEFECT REMOVAL No single method is adequate. Testing alone is insufficient. Reviews, inspections and tests combined give high efficiency, lowest costs and shortest schedules. Reviews, inspections, tests and prototypes give highest cumulative efficiency. Bad fix injection problems need special solutions. Ordinary defect removal is not adequate. Maintenance costs are cumulative, expensive and chronic.