CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 13 Software Reliability Models - Part 1

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 2 Contents Seeding and Tagging Weibull Distribution Rayleigh Distribution IEEE SW Reliability Standard Summary

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 3 Seeding and Tagging

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 4 Seeding and Tagging Introduce a given number of errors into the software -- say E of them Run standard tests, detecting D of them Compute D/E = % of errors detected Suppose D 2 = number of other errors detected Then you assume the total number of errors in the software is D 2 *E/D

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 5 Example of Seeding and Tagging 200 defects found so far Inject 20 defects Find 12 of them Therefore, assume total defects = 200 * 20 / 12 = 4000 / 12 = 333 => = 133 defects remaining By performing this analysis from time to time, you can estimate your defect density over time.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 6 Distributions Other than Exponential Weibull Distribution Rayleigh Distribution

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 7 Weibull Distribution  (shape parameter) allows reliability to increase  or to decrease   makes it equal to the exponential distribution (t)  =   t  -1 This is a useful model for trying to fit lots of different data sets, because it allows both increases and decreases.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 8 Rayleigh Distribution: Same as Weibull with  = 2 (t)  =  t  -2 Some researchers have found that this distribution fits certain software cases.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 9 IEEE Reliability Standards

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 10 The Real Need A way to manage reliability throughout the life cycle of the software This is the basis for the IEEE software reliability standard: “IEEE STD Software Reliability Metrics” & IEEE Guide (also see: IEEE STD Software Quality Assurance)

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 11 IEEE Software Reliability Standard 39 Measures applicability (1) Dobbins, James H., “SW Reliability Management,” in Handbook of SW Quality Assurance, Chapter 19.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 12 IEEE Standard Metrics Following are available for each metric – Description of use – Identification & definition of primitives – How it is implemented – How results are to be integrated – Special considerations – Special training or experience required – Specific example – Summary of benefits – Experience history – Published References

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 13 Factors Contributing to Selection of IEEE Metrics Ease of collection of primitive data Relationship between results & reliability Ease of interpretation of results Usefulness of results in management of the aspect being measured Need for measurements in each aspect of each life cycle phase Ease of implementation Cost of implementation

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 14 Three Basic Phases Predict Reliability Estimate Reliability Measure Reliability Start Coding Release Software Requirements Design Code Test Maintain Support

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 15 Some of the IEEE Metrics Unless otherwise specified, all of these are measured during development to help you assess, predict, estimate and/or control defect levels in your software.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 16 Fault Density Total Faults per 1000 Lines of Code Used to – Predict remaining faults – Assess testing sufficiency – Establish historical data You can also track this after releasing the software

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 17 Data Used to Measure Fault Density i = failure number n i = number of faults per failure N T =  n i = total faults found n f = fault density

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 18 Fault Density Formula or n f = N T / KSLOC n f =  n i / KSLOC

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 19 Graph of Fault Density

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 20 Use of Fault Density You can use fault density to estimate how close you are to finding all of the faults, and thus make decisions about whether to release software to the next development phase (or to the customer) After software release, you can continue to measure (to see how well you estimated defects remaining)

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 21 Additional Data Used to Categorize Faults d i = date of failure S i = severity of failure CL i = class of failure C i = type of fault These can help you determine which faults to focus on You may measure density separately for different classes or types or severity levels

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 22 Faults Detected (per time period) This can give you some idea of how well you are finding defects and thus how many defects are left in the software

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 23 Faults Detected Per Week

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 24 Defect Density (pre-release) This is similar to fault density, but is usually normalized by the size of the software It tells you how your software compares with other software, so you have an idea of whether yours meets your norms or expectations You can also track this after releasing the software

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 25 Defect Density Definitions (pre-release) A defect is a problem found during an inspection of the design, requirements, code, etc. i = inspection number D i = total defects found in ith inspection n = number of inspections Defect density is the total defects, normalized by the size of the software

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 26 Defect Density Equation (pre-release) n DD =  D i / KSLOC i=1

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 27 Using Defect Density To use defect density, you should track how many you find before you release the product to the next phase Over time, you can determine typical behaviors and thus meet goals or predict defect levels

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 28 Defect Density Requirements Analysis

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 29 Defect Density Software Design

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 30 Defect Density Code and Unit Test

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 31 Requirements Traceability Goals: – Identify missing requirements – Identify features that are not required Also known as “gold plating” – Measure progress in design and coding phases

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 32 Requirements Traceability Equation RT = R1 / R2 * 100% Data needed to compute this: – R1 = number of requirements traceable to specific design or code elements – R2 = total number of requirements

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 33 Requirements Traceability Variants and Usage Additional variants: – Trace to test cases – Trace from code/design/tests to requirements Usage notes: – This type of measure can be hard to keep up with and of limited utility if requirements change frequently. It works best if requirements are stable and defined.

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 34 Defect Index (for each phase) Goal: – Provide an estimate of the relative correctness of the software Primitive Data: i = phase number N i = no of defects found in phase i Further Refinement: S i = # of serious defects found in phase i M i = # of medium defects found in phase i T i = # of trivial defects found in phase i

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 35 Weighting for Defect Importance W S (typically 10) = weighting factor for serious defects W M (typically 3) = weighting factor for medium defects W T (typically 1) = weighting factor for trivial defects

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 36 Phase Defect Index W S * S i W M * M i W T * T i Pi =______ +______ + ______ N i N i N i

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 37 Use of Phase Defect Index This indicates the defect level of each phase It can be used to see if a given phase is generating too many defects and, if so, whether they are important ones

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 38 Overall Defect Index  i * P i DI =______ KSLOC

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 39 Software Maturity Index Goal: – Help determine if we are ready for delivery Primitive Data: M T = # of functions in current release F c = # of functions changed since prior release F a = # of functions added since prior release F d = # of functions deleted from prior release

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 40 Software Maturity Measure No 1: – For the most recently delivered software, count F c – For the next release, count F a, F d, M T M T - (F a + F c + F d ) MI =_______________ M T

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 41 Software Maturity Measure No 2: M T - F c SMI =_______ M T

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 42 Notes on IEEE Metrics These are indicators, not absolute measures They must be calibrated to your data in order to be of most use Despite the fact that they are over ten years old, not very many practicing software organizations have used them – Cost – Not invented here – Fear of metrics

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 43 Summary Simple models such as “seed and test” can give some concept of reliability More complex distribution models may give better results if they match the behavior of the specific type of software IEEE metrics represent one set of possible metrics for estimating and managing reliability

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 44 References IEEE STD 982.1, Software Reliability Metrics & IEEE Guide New York, Institute of Electrical and Electronics Engineers, Inc. Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS ISBN

CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 45 END OF MODULE 13