Presentation is loading. Please wait.

Presentation is loading. Please wait.

Defect Management & Metrics

Similar presentations

Presentation on theme: "Defect Management & Metrics"— Presentation transcript:

1 Defect Management & Metrics
Society for Software Quality April 24, 2010

2 Who Is This Guy? 30 years in software and hardware/software Quality
19 years in Quality & Process Management 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit Past SSQ President Corporate experience ranges from large companies such as Boeing (70,000 employees), and Sun Microsystems (45,000 employees) to very small startup companies between 250 and 13 employees. This includes a couple of turnarounds. I was the Director of Software Best Practices at Intuit for 6 years, starting the corporate SEPG. I’ve experienced several mergers and acquisitions and was directly involved in the integration of 3Com and USR and also medibuy and empactHealth. I have managed QA teams in several remote locations, including Dublin Ireland and Beijing. I have provided QA consulting for several San Diego companies, such as HNC, Stellcom, and General Atomics. Most relevant for this series of courses, I was Intuit’s Corporate Software Metrics Leader. S O C I E T Y F O R S O F T W A R E Q U A L I Y T S © 2010 – Holl Consulting

3 Primary Relevant Experience
Designed Intuit’s corporate defect management process Assisted in rollout to ~1800 engineers Guided its evolution for six years As Corporate Software Metrics Leader, fostered quantitative project management and process improvement Made mistakes and learned from them © 2010 – Holl Consulting

4 Agenda Summary of Previous Talk Defect Management Metrics
Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

5 Last time… We reviewed cultural issues related to defect management
We went over a defect management process: Policies Workflow/States Fields © 2010 – Holl Consulting

6 Data Collection At different states in the workflow, different data are collected Most data are collected at submit time Additional data are entered by the developer upon resolution And finally by the tester upon verification © 2010 – Holl Consulting

7 Agenda Summary of Previous Talk Defect Management Metrics
Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

8 When Does A Defect Count?
It depends on your goals Defect Management: To track defects for resolution in order to release products Just log functional problems found in test; deviations from the requirements that the customer will see; defects that you don’t want to fall through the cracks. Defect Analysis: To improve the process or predict If your goal is to get better (learn) or predict the future, you need to capture much more information. Without early lifecycle data (requirements, etc.) predictive models aren’t that reliable World-class companies don’t just manage defects – they analyze information. © 2010 – Holl Consulting

9 Data Collection Costs Estimate Assumptions: Time to record (18k *10m)
18,000 defects ½ the defects are corrected (9,000) Data collection time: 10 minutes (validated at HP) Average defect cost: 16 hours (data from Michael Fagan) Data collection reduces defects 5% Sampling is valid, but sampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected. Time to record (18k *10m) 3,000 hrs Time to correct (9k *16h) 144,000 hrs Defect reduction (9k * 5%) 450 defects Time savings (450 * 16h) 7,200 hrs ROI 240% This doesn’t quantify the value of being able to predict quality and schedules © 2010 – Holl Consulting

10 Simple Measures Σ (Total Resolution Effort) Number of resolved defects
Defect cost = Fix rate = Duration = Last two include queue time, so they depend on workload It’s helpful to understand the standard deviation too Bonus: By type, root cause, injection phase * Excel NetWorkDays is a useful function Number of resolved defects Period of time (week, month, etc.) Σ((Closed Date) – (Submit Date))* Number of resolved defects © 2010 – Holl Consulting

11 Relative Defect Costs If you record injection and discovery phases and effort, you can compute relative costs Example: Cost of a requirements defect = 1 Cost of a design defect = 10 Cost of a coding defect = 100 Cost of a production defect = 1000 © 2010 – Holl Consulting

12 Similar to Phase Containment
Start with matrix of injection and discovery Enter the average rework (or duration) costs in each cell (resolved defects only) Injection Phase Discovery Phase Req's Design Code Test Production Requirements 2.4 15.3 4.1 22.7 9.5 1.2 Testing 43.8 18.5 4.3 85.3 38.5 18.3 2.2 1 © 2010 – Holl Consulting

13 Similar to Phase Containment
Normalize by dividing each cell by the Injection=Discovery value Injection Phase Discovery Phase Req's Design Code Test Production Requirements 1.00 6.38 9.46 2.32 Testing 18.25 4.51 3.58 35.54 9.39 15.25 1.83 © 2010 – Holl Consulting

14 Logging Defects This is not This is
How many defects leak? This is What percent of known defects are recorded? If you only start logging defects in test, you’re missing over half of your data Try to capture data from reviews/inspections and developer unit tests © 2010 – Holl Consulting

15 Prioritizing Defects This is not about how to set priority, but…
Priority means the most important defects get resources first To measure this, we need to know the queue time (from “submit” to “work starts”) Therefore, we need an “Accepted” state, or something to note when work began Why can’t we use Submit to Verify duration? © 2010 – Holl Consulting

16 Aging Defects Time in Resolver queue Time in Verifier queue
Alternately, use durations minus effort as a proxy © 2010 – Holl Consulting

17 Defect Backlog Many groups don’t count defects left over from prior releases; they only focus on defects found and not fixed in the current project This is dangerous because a product defect backlog can grow unmanaged ALL of the open defects are released, not just the new ones When producing the report of open defects, query for all open defects in the product © 2010 – Holl Consulting

18 The Calculation Week 1 2 3 4 Total number of defects
– Number closed before the week started – Number that were opened after the week ended = Number open in the week © 2010 – Holl Consulting

19 Calculating Backlog in Excel
Query Submit Date and Closed Date © 2010 – Holl Consulting

20 Calculating Backlog in Excel
Array formula. <Shift-Ctrl-Return> This uses named ranges, where D.Num.CRs = Total number of CRs opened D.T.Closed = Range of Closed Dates D.Submit.Date = Range of Submit Dates © 2010 – Holl Consulting

21 The Result © 2010 – Holl Consulting

22 Agenda Summary of Previous Talk Defect Management Metrics
Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

23 Some Project Management Defect Metrics
Reviews & Inspections Defect Discovery & Closure Defect Density © 2010 – Holl Consulting

24 A Project Axiom The quality of product in one phase is an indicator of the quality in later phases Why? There are two answers Later work products depend on earlier ones It is an indication of the quality of the process © 2010 – Holl Consulting

25 Review Metrics Reviews & inspections have proven to be the most cost-effective way to detect and remove defects Inspect your key project artifacts (requirements, designs, code, tests, etc.) © 2010 – Holl Consulting

26 Review/Inspection Data
There are many data collected during a review or inspection The ones that tell us about quality are: Number of Operational defects discovered Number of Minor defects found per changed line of inspected doc/code Number of reviews per doc/module © 2010 – Holl Consulting

27 Project Defect Density
Consistency in categorizing inspection defects is important Have clear guidelines for Operational and Minor defects Defect Density Project = Operational Defects found by inspection Delta SLOC (or LOT) © 2010 – Holl Consulting

28 Projecting Defect Leakage
Inspected total of 5340 SLOC Found 47 operational defects Density = 47/5340 = 8.8 defects/KSLOC Project changed 18,463 SLOC Estimated defects = * 8.8 = 162 Remaining = = 115 What should you do? Inspect more? Or Test for them? © 2010 – Holl Consulting

29 Verification Use Defect Density to estimate expected yield
Similar to estimate described with reviews Defect discovery & closure Use extrapolation to predict when defect exit criteria will be met © 2010 – Holl Consulting

30 Cumulative Defects Found
Defect Discovery Total Defects Opened Weekends Cumulative Defects Found Ideal Curve! Time  © 2010 – Holl Consulting

31 Defect Discovery Curve Assumptions
Time is a proxy for “exposure” Remove weekends to smooth the curve Number of test cases run would be better, assuming Test coverage is good Not continuing to add new test case © 2010 – Holl Consulting

32 Cumulative Defects Found
Well-Behaved Curve Total Defects Opened Release Target Date Cumulative Defects Found Time  © 2010 – Holl Consulting

33 Total Defects Opened & Closed
Add: Defect Closure Total Defects Opened & Closed Remaining Open Cumulative Defects Opened Closed Time  © 2010 – Holl Consulting

34 Total Defects Opened, Closed, & To QA
Add: To QA Total Defects Opened, Closed, & To QA Opened, waiting for development fix Cumulative Defects Opened To QA Closed Turned over from development to QA, awaiting verification Time  © 2010 – Holl Consulting

35 Total Defects Opened, Closed, & To QA
Finding The Balance Total Defects Opened, Closed, & To QA What’s happening here? Cumulative Defects Dev is a bottleneck QA is a bottleneck Opened To QA Closed Time  © 2010 – Holl Consulting

36 Add: Legacy & Target Release
Total Defects Opened & Closed Inherited from previous history, targeted for or found in the project/release Cumulative Defects Total Targeted To QA Closed Time  © 2010 – Holl Consulting

37 Extrapolation to Release Date
Project Defects Release Target Date Should I be concerned? Cumulative Defects Total Targeted To QA Closed Time  © 2010 – Holl Consulting

38 Extrapolation to Release Date
Project Defects When will the project be done? Release Target Date Cumulative Defects Total Targeted To QA Closed Time  © 2010 – Holl Consulting

39 Computing in Excel Query defects into Excel
Convenient: Use named ranges for data © 2010 – Holl Consulting

40 Array formula. <Shift-Ctrl-Return>
Computing in Excel In a second sheet, create a date column and count the defects submitted and closed for each date. Array formula. <Shift-Ctrl-Return> © 2010 – Holl Consulting

41 2nd order polynomial fit
Real Life Example 2nd order polynomial fit © 2010 – Holl Consulting

42 Release Criteria: Defects
Goals: Provide good-enough quality to customers Don’t release any “bad” defects Don’t release “too many” defects Minimize support costs Customers will need support proportional to product quality © 2010 – Holl Consulting

43 Cumulative Defects Found
Defect Discovery Total Defects Opened Release Target Date Should I release? Cumulative Defects Found Time  © 2010 – Holl Consulting

44 Bad Defects Bad is a function of
Severity = the consequences of the failure Probability that it will occur The number of customers likely to experience it May also include “important” customers or functions These are often complex to calculate, so most companies just use Severity, tempered by other factors © 2010 – Holl Consulting

45 Quality Threshold No more than the following number of open defects:
Where New = defects discovered in the current project, and Old = defects discovered in previous releases Release team (stakeholders) need to determine values based on history. The goals are: Not to release any Critical defects Not to introduce “too many” new defects Manage the backlog Severity* New Old Critical Important 5 10 Moderate 30 100 Trivial 50 200 © 2010 – Holl Consulting

46 What’s Good Enough? The two market leaders – Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block TurboTax offers better videos than TaxCut … It also offers better end-of-return reviews … © 2010 – Holl Consulting

47 What’s Good Enough? TaxCut offers easier tagging of items you’d like to deal with later. It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTax’s screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected. © 2010 – Holl Consulting

48 What’s Good Enough? … if you’re choosing which program to go with for the first time this year, I’d go with TaxCut – solely because of the text errors in TurboTax. I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didn’t affect the bottom line, they did affect my confidence in the program’s makers. It’s unfortunate, because I felt TurboTax offered better features. © 2010 – Holl Consulting

49 Detroit Free Press > 1.7M Readers + web site + wire service
The article came from the Monterey Peninsula Herald © 2010 – Holl Consulting

50 Predicting Defects There are several models for predicting defects
The predictions of interest are: How many defects are going to be created? When will I find them? How long will it take to fix them? When will I be done? How good will the result be? © 2010 – Holl Consulting

51 Using Defect Density Phase Defect Density DDPhase =
At first this may not seem right, but it’s a reasonable predictor Which is more reliable? 67 specification defects per 200 pages of documentation (0.34 defects/page) 67 specification defects per 4300 SLOC (15.6 defects/KSLOC) Defects InjectedPhase Total Project Affected SLOC © 2010 – Holl Consulting

52 Using Defect Density Project/Process Defect Density
Number of defects per affected SLOC Affected SLOC can be difficult to get without a good tool. The best I’ve found is CodeReports (aka SourceROI) Release A Release B Defects 452 563 Total SLOC 8,600 43,500 Defect Density 52.6 12.9 Project Changed SLOC 5,400 4,350 Project Defect Density 83.7 129.4 © 2010 – Holl Consulting

53 Calculating DD per Phase
# Defects Injected Per Phase Project KSLOC DDPhase =  Number of Defects Injection Phase Discovery Phase Req's Design Code Test Production Requirements 46 8 26 83 19 Testing 33 22 37 2 3 16 1 Total Injected 173 69 136 Defect Density 4.94 1.97 3.89 0.06 0.03 Project SLOC = 35,000 Note: if this is only test defect data, you’re missing an opportunity to get better information. © 2010 – Holl Consulting

54 Historical Defect Density
Next Project Estimated to be 20,000 SLOC Historical Defect Density Likely to Inject Phase Containment Likely to Leak Effort to Resolve* Total Rework Requirements 4.94 99 26.6% 26 85.3 2242.2 Design 1.97 39 17.3% 7 38.5 263.3 Code 3.89 78 62.1% 48 18.3 882.9 Testing 0.06 1 81.7% 2.2 2.1 Production 0.03 100.0% 0.6 Total 218 Grand Total 3391.0 * Worst case cost (defects discovered in test) © 2010 – Holl Consulting

55 Defect Discovery & Closure
Predictions can be made in Excel by extrapolating and seeing where the discovery and close lines meet © 2010 – Holl Consulting

56 Extrapolation This method is quick and in some cases surprisingly good
But not in all cases © 2010 – Holl Consulting

57 Other Excel Extrapolations
Adding a trend line uses all the data You may only want to use some of the data Create a parallel column of the data you want to extrapolate and extrapolate only that curve © 2010 – Holl Consulting

58 Example © 2010 – Holl Consulting

59 Simple Linear Extrapolation
To create a linear extrapolation on the last few data points, start by extending the date column: © 2010 – Holl Consulting

60 Simple Linear Extrapolation
Select the cells that look linear Copy and Paste Special - Values © 2010 – Holl Consulting

61 Simple Linear Extrapolation
Then drag the cells down to the last extrapolated date Excel will do a linear extrapolation of the data © 2010 – Holl Consulting

62 Extrapolation In order for extrapolations to be reliable, the discovery rate must have peaked © 2010 – Holl Consulting

63 Done? What does it mean to be done (from a defect perspective)?
No “bad” defects remain Predictable/manageable defects remain Two categories of defects remain Those you know Those you don’t know © 2010 – Holl Consulting

64 Latent Defects Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF What needs to happen before the predictions become reliable? The defect discovery rate must flatten d f(t) dt = 0 © 2010 – Holl Consulting

65 Discovery Rate © 2010 – Holl Consulting

66 Fitting Data to a Model Fitting data to a curve helps us tell the rate is decreasing © 2010 – Holl Consulting

67 Using Excel As before, create a linear extrapolation of the last <criteria> days: “Flattening” criteria example: The discover rate must less than 5 defects per day in the past two weeks of testing © 2010 – Holl Consulting

68 Using Excel Add linear trend line Select “Display equation on chart”
© 2010 – Holl Consulting

69 Using Excel This gives the slope of the line m, where y=mx + b
In this case, m = 2.2 defects/day Not good enough © 2010 – Holl Consulting

70 Latent or Known Defect Risk
Probability of occurrence in production Risk = Severity * Probability Need to have a decent measure/estimate of probability Need a usage profile Also, frequency does not always equal value I may not use the “Restore” function very often, but it better work © 2010 – Holl Consulting

71 And Finally… Track latent/released defects carefully
They are your best indicator of Quality © 2010 – Holl Consulting

72 Release Defect Density
From: Software Quality Measurement: A Framework for Counting Problems and Defects CMU/SEI-92-TR-022 © 2010 – Holl Consulting

73 Agenda Summary of Previous Talk Defect Management Metrics
Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

74 A True Story A while back, I looked in the mirror and decided I needed to loose some weight. What is the first thing I’m going to do? Where are we? We need to step on the scale. © 2010 – Holl Consulting

75 My First Step Where are we? We need to step on the scale.
© 2010 – Holl Consulting

76 My First Step – On A Scale
If I don’t measure myself today, I won’t know If the changes I’m planning to make are taking me the right direction Which changes have the best affect How far I’ve come When I’m done It’s important to establish a baseline Where are we? We need to step on the scale. © 2010 – Holl Consulting

77 What Are The Goals? `Would you tell me, please, which way I ought to go from here?' `That depends a good deal on where you want to get to,' said the Cat. `I don't much care where--' said Alice. `Then it doesn't matter which way you go,' said the Cat. © 2010 – Holl Consulting

78 Closure Data Collection
This is where we record what we’ve learned Total Resolution Effort Resolved Injection Phase Root Cause Defect? Resolution Yes Classification, Mode Corrected Component(s) Submit reason Test case failed Pass criteria unclear Tested wrong version Setup/config wrong Data-related error Other No Correction made Duplicate Change not justified © 2010 – Holl Consulting

79 Defect Classification Models
Industry guidelines to standardize information collection The most prevalent are from: IEEE IBM Hewlett Packard © 2010 – Holl Consulting

80 HP Defect Origins, Types, and Modes
© 2010 – Holl Consulting

81 Defect Leakage Reasons
Environmental Dependency Test Not Run Test Case Error Data Dependency Inconsistent Symptoms Test Case Success Criteria Unclear Test Execution Error Test Case Results Not Verified Code Changed After Test Passed No Test Case/Coverage Configuration Management Problem Requirements Missing/Changed © 2010 – Holl Consulting

82 Using Leakage Data For each defect found in production, log a leakage reason Use the frequency (volume) to improve the related process Resist temptation to weight the frequency by something important This assumes cause & effect, and there is none This is different from root cause, where the cost to repair is related to the type of defect © 2010 – Holl Consulting

83 Specific Process Improvement Metrics
Two types of process metrics: Metrics that monitor a process (tell you the effect of changes) Basic SPC-type charts Metrics that tell you what to do Phase Containment Root Cause Analysis Test Coverage Defect Leakage Reasons Static Code Analysis - Complexity © 2010 – Holl Consulting

84 Should all projects be measured?
Pros: More data points Evaluate everything we’re doing Cons: Projects may use different processes Variation may be too great Suggestions: Don’t measure projects – measure processes Only measure your best projects Tailoring is okay, but separate out the data © 2010 – Holl Consulting

85 Hold Periodic Process Reviews
Monthly Quality Reviews focused on Process Improvement initiatives Measures & Analysis/Interpretation Lessons Learned Rules of metrics etiquette: Preview data with stakeholders = no surprises © 2010 – Holl Consulting

86 Phase Containment Identifies the lifecycle activities that leak defects Tells you where to improve defect detection efforts © 2010 – Holl Consulting

87 Root Cause Analysis Pareto the types of errors being created, weighted by cost (effort) Root Cause x Rework Hours © 2010 – Holl Consulting

88 Agenda Summary of Previous Talk Defect Management Metrics
Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

89 Defect Probabilities e e t c m f(t) = t c m =
Weibull distributions are useful engineering models, where the tail approaches zero asymptotically Weibull Probability Density Function (PDF) is: Where m determines the shape of the curve, and c is the scale parameter t c m e - f(t) = t c m m-1 e - = (Wikipedia) © 2010 – Holl Consulting

90 Different Weibull Distributions
Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions) © 2010 – Holl Consulting

91 Weibull Probability Distribution
© 2010 – Holl Consulting

92 Weibull Curves © 2010 – Holl Consulting

93 Rayleigh Probability Distribution Function
Weibull distribution with shape (m) = 2 Let x = 2t t c 2 e - f(t) = 2t c2 e t c 2 - = x c2 e 2c 2 - f(x) = x c2 e x2 4c2 - = © 2010 – Holl Consulting

94 Rayleigh Probability Distribution Function
f(x) = x c2 e x2 4c2 - © 2010 – Holl Consulting

95 Excel Formula © 2010 – Holl Consulting

96 Applying Rayleigh PDF d f(t) dt c 2
Let tm = time at which the curve reaches its peak To find the peak, we set = 0 Solving gives tm = To scale the area under the curve, multiply by K Substitute c = tm 2 d f(t) dt c 2 © 2010 – Holl Consulting

97 Fitting The Rayleigh PDF
c2 e t c 2 - f(t) = K 2t (tm 2)2 e t (tm 2) 2 - = K 2t 2t2m e - t2 = K 1 tm te - t2 2t2m f(t) = K 2 If we can estimate how many defects we’ll find (K) and when we’ve hit the peak (tm), we can create a PDF for our project! © 2010 – Holl Consulting

98 Building Your Estimate
© 2010 – Holl Consulting

99 Predicting Latent Defects
At peak, the area under the Rayleigh curve is 39.35% This means 60.65% of the defects are yet to find Example: At peak you found a cumulative total of 58 defects. Total = (58/39.5)*100=147.4 = 89 remaining. Latent Defects = K dt 1 tm te - t2 2t2m 2 Release Date 1 year? © 2010 – Holl Consulting

100 Easier that Calculus In Excel, run a column for cumulative total and a column for remaining defects From your release date you can easily estimate latent defects © 2010 – Holl Consulting

101 Cumulative Rayleigh Model
2 - CDF = 1 - e © 2010 – Holl Consulting

102 Rayleigh Assumptions The defect rate observed during the development process is positively correlated with the defect rate in the field. Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages. © 2010 – Holl Consulting

103 Exponential Models e Exponential Model (Weibull, m=1) 1 c f(t) = t -
Error detection rate Instantaneous failure rate Hazard rate In general, this is a reliability growth model © 2010 – Holl Consulting

104 Fitting A Model to Data EasyFit from
© 2010 – Holl Consulting

105 Model Summary Rayleigh is Weibull distribution with m=2
Other values of m produce other probability distributions There are many models; try several that fit “All models are wrong. Some models are useful.” George E. Box © 2010 – Holl Consulting

106 Summary A defect management program includes A balanced solution
Training & explaining Metrics that are used © 2010 – Holl Consulting

107 Chris Holl
Questions? Chris Holl Next Class: Tools for Metrics

Download ppt "Defect Management & Metrics"

Similar presentations

Ads by Google