Download presentation

Presentation is loading. Please wait.

1
**Defect Management & Metrics**

Society for Software Quality April 24, 2010

2
**Who Is This Guy? 30 years in software and hardware/software Quality**

19 years in Quality & Process Management 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit Past SSQ President Corporate experience ranges from large companies such as Boeing (70,000 employees), and Sun Microsystems (45,000 employees) to very small startup companies between 250 and 13 employees. This includes a couple of turnarounds. I was the Director of Software Best Practices at Intuit for 6 years, starting the corporate SEPG. I’ve experienced several mergers and acquisitions and was directly involved in the integration of 3Com and USR and also medibuy and empactHealth. I have managed QA teams in several remote locations, including Dublin Ireland and Beijing. I have provided QA consulting for several San Diego companies, such as HNC, Stellcom, and General Atomics. Most relevant for this series of courses, I was Intuit’s Corporate Software Metrics Leader. S O C I E T Y F O R S O F T W A R E Q U A L I Y T S © 2010 – Holl Consulting

3
**Primary Relevant Experience**

Designed Intuit’s corporate defect management process Assisted in rollout to ~1800 engineers Guided its evolution for six years As Corporate Software Metrics Leader, fostered quantitative project management and process improvement Made mistakes and learned from them © 2010 – Holl Consulting

4
**Agenda Summary of Previous Talk Defect Management Metrics**

Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

5
**Last time… We reviewed cultural issues related to defect management**

We went over a defect management process: Policies Workflow/States Fields © 2010 – Holl Consulting

6
Data Collection At different states in the workflow, different data are collected Most data are collected at submit time Additional data are entered by the developer upon resolution And finally by the tester upon verification © 2010 – Holl Consulting

7
**Agenda Summary of Previous Talk Defect Management Metrics**

Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

8
**When Does A Defect Count?**

It depends on your goals Defect Management: To track defects for resolution in order to release products Just log functional problems found in test; deviations from the requirements that the customer will see; defects that you don’t want to fall through the cracks. Defect Analysis: To improve the process or predict If your goal is to get better (learn) or predict the future, you need to capture much more information. Without early lifecycle data (requirements, etc.) predictive models aren’t that reliable World-class companies don’t just manage defects – they analyze information. © 2010 – Holl Consulting

9
**Data Collection Costs Estimate Assumptions: Time to record (18k *10m)**

18,000 defects ½ the defects are corrected (9,000) Data collection time: 10 minutes (validated at HP) Average defect cost: 16 hours (data from Michael Fagan) Data collection reduces defects 5% Sampling is valid, but sampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected. Time to record (18k *10m) 3,000 hrs Time to correct (9k *16h) 144,000 hrs Defect reduction (9k * 5%) 450 defects Time savings (450 * 16h) 7,200 hrs ROI 240% This doesn’t quantify the value of being able to predict quality and schedules © 2010 – Holl Consulting

10
**Simple Measures Σ (Total Resolution Effort) Number of resolved defects**

Defect cost = Fix rate = Duration = Last two include queue time, so they depend on workload It’s helpful to understand the standard deviation too Bonus: By type, root cause, injection phase * Excel NetWorkDays is a useful function Number of resolved defects Period of time (week, month, etc.) Σ((Closed Date) – (Submit Date))* Number of resolved defects © 2010 – Holl Consulting

11
Relative Defect Costs If you record injection and discovery phases and effort, you can compute relative costs Example: Cost of a requirements defect = 1 Cost of a design defect = 10 Cost of a coding defect = 100 Cost of a production defect = 1000 © 2010 – Holl Consulting

12
**Similar to Phase Containment**

Start with matrix of injection and discovery Enter the average rework (or duration) costs in each cell (resolved defects only) Injection Phase Discovery Phase Req's Design Code Test Production Requirements 2.4 15.3 4.1 22.7 9.5 1.2 Testing 43.8 18.5 4.3 85.3 38.5 18.3 2.2 1 © 2010 – Holl Consulting

13
**Similar to Phase Containment**

Normalize by dividing each cell by the Injection=Discovery value Injection Phase Discovery Phase Req's Design Code Test Production Requirements 1.00 6.38 9.46 2.32 Testing 18.25 4.51 3.58 35.54 9.39 15.25 1.83 © 2010 – Holl Consulting

14
**Logging Defects This is not This is**

How many defects leak? This is What percent of known defects are recorded? If you only start logging defects in test, you’re missing over half of your data Try to capture data from reviews/inspections and developer unit tests © 2010 – Holl Consulting

15
**Prioritizing Defects This is not about how to set priority, but…**

Priority means the most important defects get resources first To measure this, we need to know the queue time (from “submit” to “work starts”) Therefore, we need an “Accepted” state, or something to note when work began Why can’t we use Submit to Verify duration? © 2010 – Holl Consulting

16
**Aging Defects Time in Resolver queue Time in Verifier queue**

Alternately, use durations minus effort as a proxy © 2010 – Holl Consulting

17
Defect Backlog Many groups don’t count defects left over from prior releases; they only focus on defects found and not fixed in the current project This is dangerous because a product defect backlog can grow unmanaged ALL of the open defects are released, not just the new ones When producing the report of open defects, query for all open defects in the product © 2010 – Holl Consulting

18
**The Calculation Week 1 2 3 4 Total number of defects**

– Number closed before the week started – Number that were opened after the week ended = Number open in the week © 2010 – Holl Consulting

19
**Calculating Backlog in Excel**

Query Submit Date and Closed Date © 2010 – Holl Consulting

20
**Calculating Backlog in Excel**

Array formula. <Shift-Ctrl-Return> This uses named ranges, where D.Num.CRs = Total number of CRs opened D.T.Closed = Range of Closed Dates D.Submit.Date = Range of Submit Dates © 2010 – Holl Consulting

21
The Result © 2010 – Holl Consulting

22
**Agenda Summary of Previous Talk Defect Management Metrics**

Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

23
**Some Project Management Defect Metrics**

Reviews & Inspections Defect Discovery & Closure Defect Density © 2010 – Holl Consulting

24
A Project Axiom The quality of product in one phase is an indicator of the quality in later phases Why? There are two answers Later work products depend on earlier ones It is an indication of the quality of the process © 2010 – Holl Consulting

25
Review Metrics Reviews & inspections have proven to be the most cost-effective way to detect and remove defects Inspect your key project artifacts (requirements, designs, code, tests, etc.) © 2010 – Holl Consulting

26
**Review/Inspection Data**

There are many data collected during a review or inspection The ones that tell us about quality are: Number of Operational defects discovered Number of Minor defects found per changed line of inspected doc/code Number of reviews per doc/module © 2010 – Holl Consulting

27
**Project Defect Density**

Consistency in categorizing inspection defects is important Have clear guidelines for Operational and Minor defects Defect Density Project = Operational Defects found by inspection Delta SLOC (or LOT) © 2010 – Holl Consulting

28
**Projecting Defect Leakage**

Inspected total of 5340 SLOC Found 47 operational defects Density = 47/5340 = 8.8 defects/KSLOC Project changed 18,463 SLOC Estimated defects = * 8.8 = 162 Remaining = = 115 What should you do? Inspect more? Or Test for them? © 2010 – Holl Consulting

29
**Verification Use Defect Density to estimate expected yield**

Similar to estimate described with reviews Defect discovery & closure Use extrapolation to predict when defect exit criteria will be met © 2010 – Holl Consulting

30
**Cumulative Defects Found**

Defect Discovery Total Defects Opened Weekends Cumulative Defects Found Ideal Curve! Time © 2010 – Holl Consulting

31
**Defect Discovery Curve Assumptions**

Time is a proxy for “exposure” Remove weekends to smooth the curve Number of test cases run would be better, assuming Test coverage is good Not continuing to add new test case © 2010 – Holl Consulting

32
**Cumulative Defects Found**

Well-Behaved Curve Total Defects Opened Release Target Date Cumulative Defects Found Time © 2010 – Holl Consulting

33
**Total Defects Opened & Closed**

Add: Defect Closure Total Defects Opened & Closed Remaining Open Cumulative Defects Opened Closed Time © 2010 – Holl Consulting

34
**Total Defects Opened, Closed, & To QA**

Add: To QA Total Defects Opened, Closed, & To QA Opened, waiting for development fix Cumulative Defects Opened To QA Closed Turned over from development to QA, awaiting verification Time © 2010 – Holl Consulting

35
**Total Defects Opened, Closed, & To QA**

Finding The Balance Total Defects Opened, Closed, & To QA What’s happening here? Cumulative Defects Dev is a bottleneck QA is a bottleneck Opened To QA Closed Time © 2010 – Holl Consulting

36
**Add: Legacy & Target Release**

Total Defects Opened & Closed Inherited from previous history, targeted for or found in the project/release Cumulative Defects Total Targeted To QA Closed Time © 2010 – Holl Consulting

37
**Extrapolation to Release Date**

Project Defects Release Target Date Should I be concerned? Cumulative Defects Total Targeted To QA Closed Time © 2010 – Holl Consulting

38
**Extrapolation to Release Date**

Project Defects When will the project be done? Release Target Date Cumulative Defects Total Targeted To QA Closed Time © 2010 – Holl Consulting

39
**Computing in Excel Query defects into Excel**

Convenient: Use named ranges for data © 2010 – Holl Consulting

40
**Array formula. <Shift-Ctrl-Return>**

Computing in Excel In a second sheet, create a date column and count the defects submitted and closed for each date. Array formula. <Shift-Ctrl-Return> © 2010 – Holl Consulting

41
**2nd order polynomial fit**

Real Life Example 2nd order polynomial fit © 2010 – Holl Consulting

42
**Release Criteria: Defects**

Goals: Provide good-enough quality to customers Don’t release any “bad” defects Don’t release “too many” defects Minimize support costs Customers will need support proportional to product quality © 2010 – Holl Consulting

43
**Cumulative Defects Found**

Defect Discovery Total Defects Opened Release Target Date Should I release? Cumulative Defects Found Time © 2010 – Holl Consulting

44
**Bad Defects Bad is a function of**

Severity = the consequences of the failure Probability that it will occur The number of customers likely to experience it May also include “important” customers or functions These are often complex to calculate, so most companies just use Severity, tempered by other factors © 2010 – Holl Consulting

45
**Quality Threshold No more than the following number of open defects:**

Where New = defects discovered in the current project, and Old = defects discovered in previous releases Release team (stakeholders) need to determine values based on history. The goals are: Not to release any Critical defects Not to introduce “too many” new defects Manage the backlog Severity* New Old Critical Important 5 10 Moderate 30 100 Trivial 50 200 © 2010 – Holl Consulting

46
What’s Good Enough? The two market leaders – Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block TurboTax offers better videos than TaxCut … It also offers better end-of-return reviews … © 2010 – Holl Consulting

47
What’s Good Enough? TaxCut offers easier tagging of items you’d like to deal with later. It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTax’s screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected. © 2010 – Holl Consulting

48
What’s Good Enough? … if you’re choosing which program to go with for the first time this year, I’d go with TaxCut – solely because of the text errors in TurboTax. I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didn’t affect the bottom line, they did affect my confidence in the program’s makers. It’s unfortunate, because I felt TurboTax offered better features. © 2010 – Holl Consulting

49
**Detroit Free Press > 1.7M Readers + web site + wire service**

The article came from the Monterey Peninsula Herald © 2010 – Holl Consulting

50
**Predicting Defects There are several models for predicting defects**

The predictions of interest are: How many defects are going to be created? When will I find them? How long will it take to fix them? When will I be done? How good will the result be? © 2010 – Holl Consulting

51
**Using Defect Density Phase Defect Density DDPhase =**

At first this may not seem right, but it’s a reasonable predictor Which is more reliable? 67 specification defects per 200 pages of documentation (0.34 defects/page) 67 specification defects per 4300 SLOC (15.6 defects/KSLOC) Defects InjectedPhase Total Project Affected SLOC © 2010 – Holl Consulting

52
**Using Defect Density Project/Process Defect Density**

Number of defects per affected SLOC Affected SLOC can be difficult to get without a good tool. The best I’ve found is CodeReports (aka SourceROI) Release A Release B Defects 452 563 Total SLOC 8,600 43,500 Defect Density 52.6 12.9 Project Changed SLOC 5,400 4,350 Project Defect Density 83.7 129.4 © 2010 – Holl Consulting

53
**Calculating DD per Phase**

# Defects Injected Per Phase Project KSLOC DDPhase = Number of Defects Injection Phase Discovery Phase Req's Design Code Test Production Requirements 46 8 26 83 19 Testing 33 22 37 2 3 16 1 Total Injected 173 69 136 Defect Density 4.94 1.97 3.89 0.06 0.03 Project SLOC = 35,000 Note: if this is only test defect data, you’re missing an opportunity to get better information. © 2010 – Holl Consulting

54
**Historical Defect Density**

Next Project Estimated to be 20,000 SLOC Historical Defect Density Likely to Inject Phase Containment Likely to Leak Effort to Resolve* Total Rework Requirements 4.94 99 26.6% 26 85.3 2242.2 Design 1.97 39 17.3% 7 38.5 263.3 Code 3.89 78 62.1% 48 18.3 882.9 Testing 0.06 1 81.7% 2.2 2.1 Production 0.03 100.0% 0.6 Total 218 Grand Total 3391.0 * Worst case cost (defects discovered in test) © 2010 – Holl Consulting

55
**Defect Discovery & Closure**

Predictions can be made in Excel by extrapolating and seeing where the discovery and close lines meet © 2010 – Holl Consulting

56
**Extrapolation This method is quick and in some cases surprisingly good**

But not in all cases © 2010 – Holl Consulting

57
**Other Excel Extrapolations**

Adding a trend line uses all the data You may only want to use some of the data Create a parallel column of the data you want to extrapolate and extrapolate only that curve © 2010 – Holl Consulting

58
Example © 2010 – Holl Consulting

59
**Simple Linear Extrapolation**

To create a linear extrapolation on the last few data points, start by extending the date column: © 2010 – Holl Consulting

60
**Simple Linear Extrapolation**

Select the cells that look linear Copy and Paste Special - Values © 2010 – Holl Consulting

61
**Simple Linear Extrapolation**

Then drag the cells down to the last extrapolated date Excel will do a linear extrapolation of the data © 2010 – Holl Consulting

62
Extrapolation In order for extrapolations to be reliable, the discovery rate must have peaked © 2010 – Holl Consulting

63
**Done? What does it mean to be done (from a defect perspective)?**

No “bad” defects remain Predictable/manageable defects remain Two categories of defects remain Those you know Those you don’t know © 2010 – Holl Consulting

64
Latent Defects Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF What needs to happen before the predictions become reliable? The defect discovery rate must flatten d f(t) dt = 0 © 2010 – Holl Consulting

65
Discovery Rate © 2010 – Holl Consulting

66
Fitting Data to a Model Fitting data to a curve helps us tell the rate is decreasing © 2010 – Holl Consulting

67
Using Excel As before, create a linear extrapolation of the last <criteria> days: “Flattening” criteria example: The discover rate must less than 5 defects per day in the past two weeks of testing © 2010 – Holl Consulting

68
**Using Excel Add linear trend line Select “Display equation on chart”**

© 2010 – Holl Consulting

69
**Using Excel This gives the slope of the line m, where y=mx + b**

In this case, m = 2.2 defects/day Not good enough © 2010 – Holl Consulting

70
**Latent or Known Defect Risk**

Probability of occurrence in production Risk = Severity * Probability Need to have a decent measure/estimate of probability Need a usage profile Also, frequency does not always equal value I may not use the “Restore” function very often, but it better work © 2010 – Holl Consulting

71
**And Finally… Track latent/released defects carefully**

They are your best indicator of Quality © 2010 – Holl Consulting

72
**Release Defect Density**

From: Software Quality Measurement: A Framework for Counting Problems and Defects CMU/SEI-92-TR-022 © 2010 – Holl Consulting

73
**Agenda Summary of Previous Talk Defect Management Metrics**

Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

74
A True Story A while back, I looked in the mirror and decided I needed to loose some weight. What is the first thing I’m going to do? Where are we? We need to step on the scale. © 2010 – Holl Consulting

75
**My First Step Where are we? We need to step on the scale.**

© 2010 – Holl Consulting

76
**My First Step – On A Scale**

If I don’t measure myself today, I won’t know If the changes I’m planning to make are taking me the right direction Which changes have the best affect How far I’ve come When I’m done It’s important to establish a baseline Where are we? We need to step on the scale. © 2010 – Holl Consulting

77
What Are The Goals? `Would you tell me, please, which way I ought to go from here?' `That depends a good deal on where you want to get to,' said the Cat. `I don't much care where--' said Alice. `Then it doesn't matter which way you go,' said the Cat. © 2010 – Holl Consulting

78
**Closure Data Collection**

This is where we record what we’ve learned Total Resolution Effort Resolved Injection Phase Root Cause Defect? Resolution Yes Classification, Mode Corrected Component(s) Submit reason Test case failed Pass criteria unclear Tested wrong version Setup/config wrong Data-related error Other No Correction made Duplicate Change not justified © 2010 – Holl Consulting

79
**Defect Classification Models**

Industry guidelines to standardize information collection The most prevalent are from: IEEE IBM Hewlett Packard © 2010 – Holl Consulting

80
**HP Defect Origins, Types, and Modes**

© 2010 – Holl Consulting

81
**Defect Leakage Reasons**

Environmental Dependency Test Not Run Test Case Error Data Dependency Inconsistent Symptoms Test Case Success Criteria Unclear Test Execution Error Test Case Results Not Verified Code Changed After Test Passed No Test Case/Coverage Configuration Management Problem Requirements Missing/Changed © 2010 – Holl Consulting

82
Using Leakage Data For each defect found in production, log a leakage reason Use the frequency (volume) to improve the related process Resist temptation to weight the frequency by something important This assumes cause & effect, and there is none This is different from root cause, where the cost to repair is related to the type of defect © 2010 – Holl Consulting

83
**Specific Process Improvement Metrics**

Two types of process metrics: Metrics that monitor a process (tell you the effect of changes) Basic SPC-type charts Metrics that tell you what to do Phase Containment Root Cause Analysis Test Coverage Defect Leakage Reasons Static Code Analysis - Complexity © 2010 – Holl Consulting

84
**Should all projects be measured?**

Pros: More data points Evaluate everything we’re doing Cons: Projects may use different processes Variation may be too great Suggestions: Don’t measure projects – measure processes Only measure your best projects Tailoring is okay, but separate out the data © 2010 – Holl Consulting

85
**Hold Periodic Process Reviews**

Monthly Quality Reviews focused on Process Improvement initiatives Measures & Analysis/Interpretation Lessons Learned Rules of metrics etiquette: Preview data with stakeholders = no surprises © 2010 – Holl Consulting

86
Phase Containment Identifies the lifecycle activities that leak defects Tells you where to improve defect detection efforts © 2010 – Holl Consulting

87
Root Cause Analysis Pareto the types of errors being created, weighted by cost (effort) Root Cause x Rework Hours © 2010 – Holl Consulting

88
**Agenda Summary of Previous Talk Defect Management Metrics**

Metrics for Project Management Metrics for Process Improvement Advanced Metric Models © 2010 – Holl Consulting

89
**Defect Probabilities e e t c m f(t) = t c m =**

Weibull distributions are useful engineering models, where the tail approaches zero asymptotically Weibull Probability Density Function (PDF) is: Where m determines the shape of the curve, and c is the scale parameter t c m e - f(t) = t c m m-1 e - = (Wikipedia) © 2010 – Holl Consulting

90
**Different Weibull Distributions**

Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions) © 2010 – Holl Consulting

91
**Weibull Probability Distribution**

© 2010 – Holl Consulting

92
Weibull Curves © 2010 – Holl Consulting

93
**Rayleigh Probability Distribution Function**

Weibull distribution with shape (m) = 2 Let x = 2t t c 2 e - f(t) = 2t c2 e t c 2 - = x c2 e 2c 2 - f(x) = x c2 e x2 4c2 - = © 2010 – Holl Consulting

94
**Rayleigh Probability Distribution Function**

f(x) = x c2 e x2 4c2 - © 2010 – Holl Consulting

95
Excel Formula © 2010 – Holl Consulting

96
**Applying Rayleigh PDF d f(t) dt c 2**

Let tm = time at which the curve reaches its peak To find the peak, we set = 0 Solving gives tm = To scale the area under the curve, multiply by K Substitute c = tm 2 d f(t) dt c 2 © 2010 – Holl Consulting

97
**Fitting The Rayleigh PDF**

c2 e t c 2 - f(t) = K 2t (tm 2)2 e t (tm 2) 2 - = K 2t 2t2m e - t2 = K 1 tm te - t2 2t2m f(t) = K 2 If we can estimate how many defects we’ll find (K) and when we’ve hit the peak (tm), we can create a PDF for our project! © 2010 – Holl Consulting

98
**Building Your Estimate**

© 2010 – Holl Consulting

99
**Predicting Latent Defects**

At peak, the area under the Rayleigh curve is 39.35% This means 60.65% of the defects are yet to find Example: At peak you found a cumulative total of 58 defects. Total = (58/39.5)*100=147.4 = 89 remaining. Latent Defects = K dt 1 tm te - t2 2t2m 2 Release Date 1 year? © 2010 – Holl Consulting

100
Easier that Calculus In Excel, run a column for cumulative total and a column for remaining defects From your release date you can easily estimate latent defects © 2010 – Holl Consulting

101
**Cumulative Rayleigh Model**

2 - CDF = 1 - e © 2010 – Holl Consulting

102
Rayleigh Assumptions The defect rate observed during the development process is positively correlated with the defect rate in the field. Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages. © 2010 – Holl Consulting

103
**Exponential Models e Exponential Model (Weibull, m=1) 1 c f(t) = t -**

Error detection rate Instantaneous failure rate Hazard rate In general, this is a reliability growth model © 2010 – Holl Consulting

104
**Fitting A Model to Data EasyFit from www.mathwave.com**

© 2010 – Holl Consulting

105
**Model Summary Rayleigh is Weibull distribution with m=2**

Other values of m produce other probability distributions There are many models; try several that fit “All models are wrong. Some models are useful.” George E. Box © 2010 – Holl Consulting

106
**Summary A defect management program includes A balanced solution**

Training & explaining Metrics that are used © 2010 – Holl Consulting

107
**Chris Holl Chris_Holl@hotmail.com**

Questions? Chris Holl Next Class: Tools for Metrics

Similar presentations

OK

Validating and Improving Test-Case Effectiveness Author: Yuri Chernak Presenter: Lam, Man Tat.

Validating and Improving Test-Case Effectiveness Author: Yuri Chernak Presenter: Lam, Man Tat.

© 2018 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google