A SIMPLE VIEW OF HOW TO REDUCE TOTAL COST OF QUALITY PRESENTED BY CATHERINE OEHL COQAA Monthly Meeting December 5, 2012

AGENDA 12/5/20122 What is Quality and Why is it important What is Total Cost of Quality (TCoQ) What is the purpose and value of measuring TCoQ How to measure and reduce TCoQ Hypothetical Scenario applying TCoQ Q&A

QUALITY DEFINED Definition: A measure of excellence or a state of being free from defects, deficiencies and significant variations. It is brought about by strict and consistent commitment to certain standards that achieve uniformity of a product in order to satisfy specific customer or user requirements. ISO standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs." If an automobile company finds a defect in one of their cars and makes a product recall, customer reliability and therefore production will decrease because trust will be lost in the car's quality. 12/5/20123 Source:

CONSEQUENCES OF POOR QUALITY 12/5/20124

COMMON STATE OF QUALITY o Current projects spend 40-50% of the effort on avoidable rework o About 80% of avoidable rework comes from 20% of the defects o About 80% of defects comes from 20% of the modules and about half of the modules are defects free o 90% of the downtime comes from at most 10% of the defects o Peer Reviews catch 60% of the defects o Perspective-based reviews catch 35% more defects than non directed reviews o Disciplined personal practices can reduce defect introduction rates by up to 75% o While several organizations measure and track application total cost of ownership (TCO), very few measure the total cost of quality 12/5/20125

WHAT IS TCOQ Total Cost of Quality (TCoQ) consists of the costs associated with preventing, assessing, finding and correcting defective work. Note that the TCoQ analysis looks only at our (producers) costs and not the customers cost due to poor quality. 12/5/20126

QUALITY COSTS OF TCOQ Prevention cost + Inspection cost + Internal Failure cost + External Failure cost Prevention costs arise from efforts to keep errors or defects from occurring at all Inspection costs arise from detecting errors/defects via inspections, tests, audits Internal failure costs arise from defects caught internally and dealt with by discarding or repairing the defective items External failure costs arise from defects that actually reach customers. Often other costs are incurred in addition to correcting the failure. 12/5/20127

PURPOSE AND VALUE OF TCOQ Addresses and measures the following questions: What is the state of quality of your applications? How does quality impact your business? How is quality measured? When are errors/defects introduced? Why? When are errors/defects corrected? What are the costs? What are the processes that impact quality? How do you quantify the effectiveness of each process? How much do failures cost? What is the root cause of the failure? 12/5/20128

WHEN AND WHERE ARE DEFECTS INTRODUCED Source: Writing Testable Requirements, Dick Bender 12/5/20129 Source: NIST Software Quality Study

WHEN AND WHERE ARE DEFECTS DETECTED 12/5/ Source: NIST Software Quality Study

COST TO CORRECT DEFECTS 12/5/ Source: B. Boehm and V. Basili, Software Defect Reduction Top 10 List, IEEE Computer Cost to remove defects is 100 times more

THE QUALITY LEVER 12/5/ Source: Prevention / Reviews Appraisal / Inspection Failures / Reaction Cost Savings Greatest Leverage Good Leverage Least Leverage

PREVENTIVE COSTS GOAL: Prevent defects before they happen (Work done right the first time) Recommended Activities: –Formal Requirements Gathering Processes –Requirements Analysis and Management –Early Prototyping –Usability Analysis –Static Testing –Peer Reviews –Release and Configuration Management Processes –Standards for Test Environments –Quality Training –Accurate Internal Communication 12/5/201213

INSPECTION COSTS GOAL: Detect the Errors (Find them early) Recommended Activities: –Code inspections –Design reviews –Glass box and black box testing –Test Automation –Usability Testing –Pre-release / User Acceptance Testing –Structured Personal reviews focused on verification and validation 12/05/201214

INTERNAL FAILURE COSTS GOAL: Report and fix defects before production release (Fix and Re-Test) Incurred Activities: Diagnose Defects Defect Fixes Defect Management Re-Test and Regression Testing Rework code and recompile Wait states 12/5/201215

EXTERNAL FAILURE COSTS GOAL: Reduce and remediate impact to customers (Apologize and recover) Incurred Activities: Technical Support Calls Recalls Customer Service Scrap Overtime Regulatory / Legal Costs 12/5/201216

HYPOTHETICAL SCENARIO Suppose we have a software product that has one production release per quarter. On average each release contains 1000 must-fix defects that will be identified and repaired. We will assume developers catch 25% of defects during unit and integration testing at a cost of $10/defect 12/5/201217

1 - NO INSPECTION TESTING 12/5/ Inspection Testing Costs Non Formal Testing Staff$0 Infrastructure$0 Tools$0 Total Investment$0 Development Must-Fix Defects Found250 Fix Cost (Internal Failure - $10/defect)$2,500 Testing (Pre-Release) Must-Fix Defects Found0 Fix Cost (Internal Failure - $100/defect)$0 Customer Support (Production) Must-Fix Defects Reported750 Fix Cost (External Failure - $1000/defect)$750,000 Cost of Quality Conformance (Inspection Costs)$0 NonConformance (Internal Failures + External Failures)$752,500 TCoQ$752,500 ROIN/A

2 - MANUAL INSPECTION TESTING 12/5/ Inspection Testing Costs Non Formal Testing Manual Testing Staff$0$60,000 Infrastructure$0$10,000 Tools$0 Total Investment$0$70,000 Development Must-Fix Defects Found250 Fix Cost (Internal Failure - $10/defect)$2,500 Testing (Pre-Release) Must-Fix Defects Found0350 Fix Cost (Internal Failure - $100/defect)$0$35,000 Customer Support (Production) Must-Fix Defects Reported Fix Cost (External Failure - $1000/defect)$750,000$400,000 Cost of Quality Conformance (Inspection Costs)$0$70,000 NonConformance (Internal Failures + External Failures)$752,500$437,500 TCoQ$752,500$507,500 ROIN/A350%

3 - MANUAL WITH AUTOMATED INSPECTION TESTING 12/5/ Inspection Testing Costs Non Formal Testing Manual Testing Automated Testing Staff$0$60,000 Infrastructure$0$10,000 Tools$0 $12,500 Total Investment$0$70,000$82,500 Development Must-Fix Defects Found250 Fix Cost (Internal Failure - $10/defect)$2,500 Testing (Pre-Release) Must-Fix Defects Found Fix Cost (Internal Failure - $100/defect)$0$35,000$50,000 Customer Support (Production) Must-Fix Defects Reported Fix Cost (External Failure - $1000/defect)$750,000$400,000$250,000 Cost of Quality Conformance (Inspection Costs)$0$70,000$82,500 NonConformance (Internal Failures + External Failures)$752,500$437,500$302,500 TCoQ$752,500$507,500$385,000 ROIN/A350%445% What if we added some simple Prevention activities like Static Reviews, Peer Reviews, Change Management ?

4 - ADDING PREVENTION ACTIVITIES 12/5/ Prevention Costs Resource Costs Static Reviews (10 people every 2 weeks reviewing 5 sets of documents x $75/hr) $97,500 Peer Reviews (2 developers every 2 weeks reviewing 10 modules x $75) $39,000 Change Management (10% Configuration Mgr & 10% Requirements Lead & 10% Test Lead) $46,800 Inspection Testing CostsNon Formal TestingManual TestingAutomated Testing Staff$0$60,000 Infrastructure$0$10,000 Tools$0 $12,500 Total Investment (Prevention & Inspection costs)$0$70,000$82,500$265,800 Development Must-Fix Defects Found Fix Cost (Internal Failure - $10/defect)$2,500 $5,000 Testing (Pre-Release) Must-Fix Defects Found Fix Cost (Internal Failure - $100/defect)$0$35,000$50,000$35,000 Customer Support (Production) Must-Fix Defects Reported Fix Cost (External Failure - $1000/defect)$750,000$400,000$250,000$50,000 Cost of Quality Conformance (Inspection Costs)$0$70,000$82,500$265,800 NonConformance (Internal Failures + External Failures)$752,500$437,500$302,500$90,000 TCoQ$752,500$507,500$385,000$355,800 ROIN/A350%445%57%

SUMMARY Quality creates value and reduces risk exposure The cost of quality isnt the price of creating a quality product or service. Its the cost of NOT creating a quality product or service. Errors / defects are often introduced early and typically found later Total Cost of Quality (TCoQ) consists of the costs associated with preventing, assessing, finding and correcting defective work. The central theme of quality improvement is that larger investments in prevention drive even larger savings in quality-related failures and Inspection efforts. Any Quality improvement effort should use TCoQ to quantify the effectiveness of the process. 12/5/201222

CONTACT INFO AND Q&A Catherine Oehl LinkedIn Profile: 12/5/201223

RESOURCES & REFERENCES Software Defect Reduction Top 10 List by Barry Boehm and Victor Basili Using Cost Benefit Analysis to Develop Software Process Improvement Strategies, Data Analysis Center of Software Traditional P-A-F method by Juran (1951) and Feigenbaum (1956) – Prevention Appraisal and Failure costs Phil Crosbys Model from the book Quality is Free (1979) – conformance to requirements – cost of conformance (done Right) and non-conformance (done Wrong) Process Cost Model by Ross (1977) quality quality only-failure-costs/ only-failure-costs/ 12/5/201224

SUGGESTED TCOQ BENCHMARK MEASUREMENTS Process (Productivity & Prevention) –Effectiveness of Test Processes Release Mgt, Configuration Mgt –Efficiency of Test Processes Defect Mgt, Test Environment Mgt, Test Data Mgt –Consistency of Test Activities Test Planning, Test Estimation, Test Execution Schedule –Testing Planning Duration of lifecycle phases Duration of Test Phase/Stage –Test Estimating/Projections Planning Estimate Component Based Estimates 12/5/201225

COMMON MEASUREMENTS OF FAILURES Failures –Volume of Defects Open, Closed, New, Test Phase/Stage Contained, Defect Aging (# of days open) –Severity of Defects Criticals, Highs, Mediums, Lows, Problem Areas, Low to all reported ratio 12/5/201226

SIMPLE ACTUAL QUALITY COSTS Build Test Mgt Costs (P+I+IF) –Manual Test Hour Costs –Automated Test Hour Costs –Test Hour Costs per Test Phase/Stage –% of Total Build Costs Breakfix/Maintenance Costs (EF) 12/5/201227

VIEW OF TCOQ IN TERMS OF QA 12/5/201228