Presentation is loading. Please wait.

Presentation is loading. Please wait.

Defect Management & Metrics Society for Software Quality April 24, 2010.

Similar presentations


Presentation on theme: "Defect Management & Metrics Society for Software Quality April 24, 2010."— Presentation transcript:

1 Defect Management & Metrics Society for Software Quality April 24, 2010

2 SSQ Presentation © 2010 – Holl Consulting Who Is This Guy? 30 years in software and hardware/software Quality 19 years in Quality & Process Management 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit Past SSQ President Q U A L I Y T Q S S

3 SSQ Presentation © 2010 – Holl Consulting Primary Relevant Experience Designed Intuit’s corporate defect management process Assisted in rollout to ~1800 engineers Guided its evolution for six years As Corporate Software Metrics Leader, fostered quantitative project management and process improvement Made mistakes and learned from them

4 SSQ Presentation © 2010 – Holl Consulting Agenda Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

5 SSQ Presentation © 2010 – Holl Consulting Last time… We reviewed cultural issues related to defect management We went over a defect management process: –Policies –Workflow/States –Fields

6 SSQ Presentation © 2010 – Holl Consulting Data Collection At different states in the workflow, different data are collected Most data are collected at submit time Additional data are entered by the developer upon resolution And finally by the tester upon verification

7 SSQ Presentation © 2010 – Holl Consulting Agenda Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

8 SSQ Presentation © 2010 – Holl Consulting When Does A Defect Count? It depends on your goals Defect Management: To track defects for resolution in order to release products –Just log functional problems found in test; deviations from the requirements that the customer will see; defects that you don’t want to fall through the cracks. Defect Analysis: To improve the process or predict –If your goal is to get better (learn) or predict the future, you need to capture much more information. –Without early lifecycle data (requirements, etc.) predictive models aren’t that reliable World-class companies don’t just manage defects – they analyze information.

9 SSQ Presentation © 2010 – Holl Consulting Data Collection Costs Estimate Assumptions: –18,000 defects –½ the defects are corrected (9,000) –Data collection time: 10 minutes (validated at HP) –Average defect cost: 16 hours (data from Michael Fagan) –Data collection reduces defects 5% Time to record (18k *10m)3,000 hrs Time to correct (9k *16h)144,000 hrs Defect reduction (9k * 5%)450 defects Time savings (450 * 16h)7,200 hrs ROI240% Sampling is valid, but sampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected. This doesn’t quantify the value of being able to predict quality and schedules

10 SSQ Presentation © 2010 – Holl Consulting Simple Measures Defect cost = Fix rate = Duration = Last two include queue time, so they depend on workload It’s helpful to understand the standard deviation too Bonus: By type, root cause, injection phase * Excel NetWorkDays is a useful function Σ (Total Resolution Effort) Number of resolved defects Period of time (week, month, etc.) Σ((Closed Date) – (Submit Date))* Number of resolved defects

11 SSQ Presentation © 2010 – Holl Consulting Relative Defect Costs If you record injection and discovery phases and effort, you can compute relative costs Example: –Cost of a requirements defect = 1 –Cost of a design defect = 10 –Cost of a coding defect = 100 –Cost of a production defect = 1000

12 SSQ Presentation © 2010 – Holl Consulting Similar to Phase Containment Start with matrix of injection and discovery Enter the average rework (or duration) costs in each cell (resolved defects only) Injection Phase Discovery Phase Req'sDesignCodeTestProduction Requirements2.4 Design Code Testing Production

13 SSQ Presentation © 2010 – Holl Consulting Similar to Phase Containment Normalize by dividing each cell by the Injection=Discovery value Injection Phase Discovery Phase Req'sDesignCodeTestProduction Requirements1.00 Design Code Testing Production

14 SSQ Presentation © 2010 – Holl Consulting Logging Defects This is not How many defects leak? This is What percent of known defects are recorded? If you only start logging defects in test, you’re missing over half of your data Try to capture data from reviews/inspections and developer unit tests

15 SSQ Presentation © 2010 – Holl Consulting Prioritizing Defects This is not about how to set priority, but… Priority means the most important defects get resources first To measure this, we need to know the queue time (from “submit” to “work starts”) Therefore, we need an “Accepted” state, or something to note when work began Why can’t we use Submit to Verify duration?

16 SSQ Presentation © 2010 – Holl Consulting Aging Defects Time in Resolver queue Time in Verifier queue Alternately, use durations minus effort as a proxy

17 SSQ Presentation © 2010 – Holl Consulting Defect Backlog Many groups don’t count defects left over from prior releases; they only focus on defects found and not fixed in the current project This is dangerous because a product defect backlog can grow unmanaged ALL of the open defects are released, not just the new ones When producing the report of open defects, query for all open defects in the product

18 SSQ Presentation © 2010 – Holl Consulting The Calculation Week Total number of defects – Number closed before the week started – Number that were opened after the week ended = Number open in the week

19 SSQ Presentation © 2010 – Holl Consulting Calculating Backlog in Excel Query Submit Date and Closed Date

20 SSQ Presentation © 2010 – Holl Consulting Calculating Backlog in Excel This uses named ranges, where D.Num.CRs= Total number of CRs opened D.T.Closed= Range of Closed Dates D.Submit.Date= Range of Submit Dates Array formula.

21 SSQ Presentation © 2010 – Holl Consulting The Result

22 SSQ Presentation © 2010 – Holl Consulting Agenda Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

23 SSQ Presentation © 2010 – Holl Consulting Some Project Management Defect Metrics Reviews & Inspections Defect Discovery & Closure Defect Density

24 SSQ Presentation © 2010 – Holl Consulting A Project Axiom The quality of product in one phase is an indicator of the quality in later phases Why? –There are two answers

25 SSQ Presentation © 2010 – Holl Consulting Review Metrics Reviews & inspections have proven to be the most cost- effective way to detect and remove defects Inspect your key project artifacts (requirements, designs, code, tests, etc.)

26 SSQ Presentation © 2010 – Holl Consulting Review/Inspection Data There are many data collected during a review or inspection The ones that tell us about quality are: –Number of Operational defects discovered –Number of Minor defects found per changed line of inspected doc/code Number of reviews per doc/module

27 SSQ Presentation © 2010 – Holl Consulting Project Defect Density Consistency in categorizing inspection defects is important –Have clear guidelines for Operational and Minor defects Defect Density Project = Operational Defects found by inspection Delta SLOC (or LOT)

28 SSQ Presentation © 2010 – Holl Consulting Projecting Defect Leakage Inspected total of 5340 SLOC Found 47 operational defects Density = 47/5340 = 8.8 defects/KSLOC Project changed 18,463 SLOC Estimated defects = * 8.8 = 162 Remaining = = 115 What should you do? Inspect more? Or Test for them?

29 SSQ Presentation © 2010 – Holl Consulting Verification Use Defect Density to estimate expected yield –Similar to estimate described with reviews Defect discovery & closure Use extrapolation to predict when defect exit criteria will be met

30 SSQ Presentation © 2010 – Holl Consulting Defect Discovery Time  Cumulative Defects Found Total Defects Opened Ideal Curve! Weekends

31 SSQ Presentation © 2010 – Holl Consulting Defect Discovery Curve Assumptions Time is a proxy for “exposure” –Remove weekends to smooth the curve Number of test cases run would be better, assuming –Test coverage is good –Not continuing to add new test case

32 SSQ Presentation © 2010 – Holl Consulting Well-Behaved Curve Time  Cumulative Defects Found Total Defects Opened Release Target Date

33 SSQ Presentation © 2010 – Holl Consulting Add: Defect Closure Time  Cumulative Defects Total Defects Opened & Closed Opened Closed Remaining Open

34 SSQ Presentation © 2010 – Holl Consulting Add: To QA Time  Cumulative Defects Total Defects Opened, Closed, & To QA Opened To QA Closed Opened, waiting for development fix Turned over from development to QA, awaiting verification

35 SSQ Presentation © 2010 – Holl Consulting Finding The Balance Time  Cumulative Defects Total Defects Opened, Closed, & To QA Opened To QA Closed What’s happening here? QA is a bottleneck Dev is a bottleneck

36 SSQ Presentation © 2010 – Holl Consulting Add: Legacy & Target Release Time  Cumulative Defects Total Defects Opened & Closed Total Targeted To QA Closed Inherited from previous history, targeted for or found in the project/release

37 SSQ Presentation © 2010 – Holl Consulting Extrapolation to Release Date Time  Cumulative Defects Project Defects Total Targeted To QA Closed Release Target Date Should I be concerned?

38 SSQ Presentation © 2010 – Holl Consulting Extrapolation to Release Date Time  Cumulative Defects Project Defects Total Targeted To QA Closed Release Target Date When will the project be done?

39 SSQ Presentation © 2010 – Holl Consulting Computing in Excel Query defects into Excel Convenient: Use named ranges for data

40 SSQ Presentation © 2010 – Holl Consulting Computing in Excel In a second sheet, create a date column and count the defects submitted and closed for each date. Array formula.

41 SSQ Presentation © 2010 – Holl Consulting Real Life Example 2 nd order polynomial fit

42 SSQ Presentation © 2010 – Holl Consulting Release Criteria: Defects Goals: –Provide good-enough quality to customers Don’t release any “bad” defects Don’t release “too many” defects –Minimize support costs Customers will need support proportional to product quality

43 SSQ Presentation © 2010 – Holl Consulting Defect Discovery Time  Cumulative Defects Found Total Defects Opened Release Target Date Should I release?

44 SSQ Presentation © 2010 – Holl Consulting Bad Defects Bad is a function of –Severity = the consequences of the failure –Probability that it will occur –The number of customers likely to experience it –May also include “important” customers or functions These are often complex to calculate, so most companies just use Severity, tempered by other factors

45 SSQ Presentation © 2010 – Holl Consulting Quality Threshold No more than the following number of open defects: –Where New = defects discovered in the current project, and –Old = defects discovered in previous releases Severity*NewOld Critical00 Important510 Moderate30100 Trivial50200 Release team (stakeholders) need to determine values based on history. The goals are: Not to release any Critical defects Not to introduce “too many” new defects Manage the backlog

46 SSQ Presentation © 2010 – Holl Consulting What’s Good Enough? The two market leaders – Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block It also offers better end- of-return reviews … TurboTax offers better videos than TaxCut …

47 SSQ Presentation © 2010 – Holl Consulting What’s Good Enough? TaxCut offers easier tagging of items you’d like to deal with later. It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTax’s screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected.

48 SSQ Presentation © 2010 – Holl Consulting What’s Good Enough? … if you’re choosing which program to go with for the first time this year, I’d go with TaxCut – solely because of the text errors in TurboTax. I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didn’t affect the bottom line, they did affect my confidence in the program’s makers. It’s unfortunate, because I felt TurboTax offered better features.

49 SSQ Presentation © 2010 – Holl Consulting Detroit Free Press > 1.7M Readers + web site + wire service The article came from the Monterey Peninsula Herald

50 SSQ Presentation © 2010 – Holl Consulting Predicting Defects There are several models for predicting defects The predictions of interest are: –How many defects are going to be created? –When will I find them? –How long will it take to fix them? –When will I be done? –How good will the result be?

51 SSQ Presentation © 2010 – Holl Consulting Using Defect Density Phase Defect Density DD Phase = At first this may not seem right, but it’s a reasonable predictor Which is more reliable? –67 specification defects per 200 pages of documentation (0.34 defects/page) –67 specification defects per 4300 SLOC (15.6 defects/KSLOC) Defects Injected Phase Total Project Affected SLOC

52 SSQ Presentation © 2010 – Holl Consulting Using Defect Density Project/Process Defect Density –Number of defects per affected SLOC –Affected SLOC can be difficult to get without a good tool. –The best I’ve found is CodeReports (aka SourceROI) –http://www.polariguana.com/sourceroi.php Release ARelease B Defects Total SLOC 8,60043,500 Defect Density Project Changed SLOC 5,4004,350 Project Defect Density

53 SSQ Presentation © 2010 – Holl Consulting Calculating DD per Phase # Defects Injected Per Phase Project KSLOC Number of DefectsInjection Phase Discovery PhaseReq'sDesignCodeTestProduction Requirements Design Code Testing Production Total Injected Defect Density Project SLOC = 35,000 Note: if this is only test defect data, you’re missing an opportunity to get better information. DD Phase =

54 SSQ Presentation © 2010 – Holl Consulting Next Project Estimated to be 20,000 SLOC Historical Defect Density Likely to Inject Phase Containment Likely to Leak Effort to Resolve* Total Rework Requirements % Design % Code % Testing % Production %110.6 Total218Grand Total * Worst case cost (defects discovered in test)

55 SSQ Presentation © 2010 – Holl Consulting Defect Discovery & Closure Predictions can be made in Excel by extrapolating and seeing where the discovery and close lines meet

56 SSQ Presentation © 2010 – Holl Consulting Extrapolation This method is quick and in some cases surprisingly good But not in all cases

57 SSQ Presentation © 2010 – Holl Consulting Other Excel Extrapolations Adding a trend line uses all the data You may only want to use some of the data Create a parallel column of the data you want to extrapolate and extrapolate only that curve

58 SSQ Presentation © 2010 – Holl Consulting Example

59 SSQ Presentation © 2010 – Holl Consulting Simple Linear Extrapolation To create a linear extrapolation on the last few data points, start by extending the date column:

60 SSQ Presentation © 2010 – Holl Consulting Simple Linear Extrapolation Select the cells that look linear Copy and Paste Special - Values

61 SSQ Presentation © 2010 – Holl Consulting Simple Linear Extrapolation Then drag the cells down to the last extrapolated date Excel will do a linear extrapolation of the data

62 SSQ Presentation © 2010 – Holl Consulting Extrapolation In order for extrapolations to be reliable, the discovery rate must have peaked

63 SSQ Presentation © 2010 – Holl Consulting Done? What does it mean to be done (from a defect perspective)? –No “bad” defects remain –Predictable/manageable defects remain Two categories of defects remain –Those you know –Those you don’t know

64 SSQ Presentation © 2010 – Holl Consulting Latent Defects Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF What needs to happen before the predictions become reliable? –The defect discovery rate must flatten d f ( t ) dt = 0

65 SSQ Presentation © 2010 – Holl Consulting Discovery Rate

66 SSQ Presentation © 2010 – Holl Consulting Fitting Data to a Model Fitting data to a curve helps us tell the rate is decreasing

67 SSQ Presentation © 2010 – Holl Consulting Using Excel As before, create a linear extrapolation of the last days: “Flattening” criteria example: –The discover rate must less than 5 defects per day in the past two weeks of testing

68 SSQ Presentation © 2010 – Holl Consulting Using Excel Add linear trend line Select “Display equation on chart”

69 SSQ Presentation © 2010 – Holl Consulting Using Excel This gives the slope of the line m, where y=mx + b –In this case, m = 2.2 defects/day –Not good enough

70 SSQ Presentation © 2010 – Holl Consulting Latent or Known Defect Risk Probability of occurrence in production –Risk = Severity * Probability Need to have a decent measure/estimate of probability –Need a usage profile Also, frequency does not always equal value –I may not use the “Restore” function very often, but it better work

71 SSQ Presentation © 2010 – Holl Consulting And Finally… Track latent/released defects carefully They are your best indicator of Quality

72 SSQ Presentation © 2010 – Holl Consulting Release Defect Density From: Software Quality Measurement: A Framework for Counting Problems and Defects CMU/SEI-92-TR-022 ocuments/92.reports/pdf/tr pdf

73 SSQ Presentation © 2010 – Holl Consulting Agenda Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

74 SSQ Presentation © 2010 – Holl Consulting A True Story A while back, I looked in the mirror and decided I needed to loose some weight. What is the first thing I’m going to do?

75 SSQ Presentation © 2010 – Holl Consulting My First Step

76 SSQ Presentation © 2010 – Holl Consulting My First Step – On A Scale If I don’t measure myself today, I won’t know –If the changes I’m planning to make are taking me the right direction –Which changes have the best affect –How far I’ve come –When I’m done It’s important to establish a baseline

77 SSQ Presentation © 2010 – Holl Consulting What Are The Goals? `Would you tell me, please, which way I ought to go from here?' `That depends a good deal on where you want to get to,' said the Cat. `I don't much care where--' said Alice. `Then it doesn't matter which way you go,' said the Cat.

78 SSQ Presentation © 2010 – Holl Consulting Closure Data Collection Submit reason Test case failed Pass criteria unclear Tested wrong version Setup/config wrong Data-related error Other No Correction made Duplicate Change not justified Injection Phase Root Cause Defect? Resolution Yes Classification, Mode Corrected Component(s) This is where we record what we’ve learned Total Resolution Effort Resolved

79 SSQ Presentation © 2010 – Holl Consulting Defect Classification Models Industry guidelines to standardize information collection The most prevalent are from: –IEEE –IBM –Hewlett Packard

80 SSQ Presentation © 2010 – Holl Consulting HP Defect Origins, Types, and Modes

81 SSQ Presentation © 2010 – Holl Consulting Defect Leakage Reasons 1.Environmental Dependency 2.Test Not Run 3.Test Case Error 4.Data Dependency 5.Inconsistent Symptoms 6.Test Case Success Criteria Unclear 7.Test Execution Error 8.Test Case Results Not Verified 9.Code Changed After Test Passed 10.No Test Case/Coverage 11.Configuration Management Problem 12.Requirements Missing/Changed

82 SSQ Presentation © 2010 – Holl Consulting Using Leakage Data For each defect found in production, log a leakage reason Use the frequency (volume) to improve the related process Resist temptation to weight the frequency by something important –This assumes cause & effect, and there is none –This is different from root cause, where the cost to repair is related to the type of defect

83 SSQ Presentation © 2010 – Holl Consulting Specific Process Improvement Metrics Two types of process metrics: 1.Metrics that monitor a process (tell you the effect of changes) Basic SPC-type charts 2.Metrics that tell you what to do Phase Containment Root Cause Analysis Test Coverage Defect Leakage Reasons Static Code Analysis - Complexity

84 SSQ Presentation © 2010 – Holl Consulting Should all projects be measured? Pros: –More data points –Evaluate everything we’re doing Cons: –Projects may use different processes –Variation may be too great Suggestions: –Don’t measure projects – measure processes –Only measure your best projects –Tailoring is okay, but separate out the data

85 SSQ Presentation © 2010 – Holl Consulting Hold Periodic Process Reviews Monthly Quality Reviews focused on –Process Improvement initiatives –Measures & Analysis/Interpretation –Lessons Learned Rules of metrics etiquette: –Preview data with stakeholders = no surprises

86 SSQ Presentation © 2010 – Holl Consulting Phase Containment Identifies the lifecycle activities that leak defects Tells you where to improve defect detection efforts

87 SSQ Presentation © 2010 – Holl Consulting Root Cause Analysis Pareto the types of errors being created, weighted by cost (effort) Root Cause x Rework Hours

88 SSQ Presentation © 2010 – Holl Consulting Agenda Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models

89 SSQ Presentation © 2010 – Holl Consulting Defect Probabilities Weibull distributions are useful engineering models, where the tail approaches zero asymptotically Weibull Probability Density Function (PDF) is: Where m determines the shape of the curve, and c is the scale parameter tctc mtmt m e tctc m - f (t) = tctc mcmc m-1 e tctc m - = (Wikipedia)

90 SSQ Presentation © 2010 – Holl Consulting Different Weibull Distributions Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions)

91 SSQ Presentation © 2010 – Holl Consulting Weibull Probability Distribution

92 SSQ Presentation © 2010 – Holl Consulting Weibull Curves

93 SSQ Presentation © 2010 – Holl Consulting Rayleigh Probability Distribution Function Weibull distribution with shape (m) = 2 Let x = 2t tctc 2t2t 2 e tctc 2 - f (t) = 2tc22tc2 e tctc 2 - = xc2xc2 e x2cx2c 2 - f (x) = xc2xc2 e x24c2x24c2 - =

94 SSQ Presentation © 2010 – Holl Consulting Rayleigh Probability Distribution Function xc2xc2 e x24c2x24c2 - f (x) =

95 SSQ Presentation © 2010 – Holl Consulting Excel Formula

96 SSQ Presentation © 2010 – Holl Consulting Applying Rayleigh PDF Let t m = time at which the curve reaches its peak To find the peak, we set = 0 Solving gives t m = To scale the area under the curve, multiply by K Substitute c = t m 2 d f ( t ) dt c2c2

97 SSQ Presentation © 2010 – Holl Consulting Fitting The Rayleigh PDF 2tc22tc2 e tctc 2 - f (t) = K 2t ( t m 2 ) 2 e t (t m 2) 2 - = K 2t2t2m2t2t2m e - t 2 2t 2 m = K 1tm1tm te - t 2 2t 2 m f (t) = K 2 If we can estimate how many defects we’ll find ( K ) and when we’ve hit the peak ( t m ), we can create a PDF for our project!

98 SSQ Presentation © 2010 – Holl Consulting Building Your Estimate

99 SSQ Presentation © 2010 – Holl Consulting Predicting Latent Defects At peak, the area under the Rayleigh curve is 39.35% This means 60.65% of the defects are yet to find Example: –At peak you found a cumulative total of 58 defects. –Total = (58/39.5)*100=147.4 – = 89 remaining. Latent Defects = Kdt 1tm1tm te - t 2 2t 2 m 2 Release Date 1 year?

100 SSQ Presentation © 2010 – Holl Consulting Easier that Calculus In Excel, run a column for cumulative total and a column for remaining defects From your release date you can easily estimate latent defects

101 SSQ Presentation © 2010 – Holl Consulting Cumulative Rayleigh Model tctc 2 - CDF = 1 - e

102 SSQ Presentation © 2010 – Holl Consulting Rayleigh Assumptions The defect rate observed during the development process is positively correlated with the defect rate in the field. Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages.

103 SSQ Presentation © 2010 – Holl Consulting Exponential Models Exponential Model (Weibull, m=1) 1c1c e tctc - f (t) = Error detection rate Instantaneous failure rate Hazard rate In general, this is a reliability growth model

104 SSQ Presentation © 2010 – Holl Consulting Fitting A Model to Data EasyFit from

105 SSQ Presentation © 2010 – Holl Consulting Model Summary Rayleigh is Weibull distribution with m=2 Other values of m produce other probability distributions There are many models; try several that fit “All models are wrong. Some models are useful.” –George E. Box

106 SSQ Presentation © 2010 – Holl Consulting Summary A defect management program includes –A balanced solution –Training & explaining –Metrics that are used

107 Questions ? Chris Holl Next Class: Tools for Metrics


Download ppt "Defect Management & Metrics Society for Software Quality April 24, 2010."

Similar presentations


Ads by Google