Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 SMU CSE 8314 Software Measurement and Quality Engineering Module 13 Software Reliability Models - Part 1

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Contents  Seeding and Tagging  Weibull Distribution  Rayleigh Distribution  IEEE SW Reliability Standard  Summary

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 Seeding and Tagging

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Seeding and Tagging  Introduce a given number of errors into the software -- say E of them  Run standard tests, detecting D of them  Compute D/E = % of errors detected  Suppose D 2 = number of other errors detected  Then you assume the total number of errors in the software is D 2 *E/D

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Example of Seeding and Tagging  200 defects found so far  Inject 20 defects  Find 12 of them  Therefore, assume total defects = 200 * 20 / 12 = 4000 / 12 = 333 => = 133 defects remaining By performing this analysis from time to time, you can estimate your defect density over time.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 Distributions Other than Exponential Weibull Distribution Rayleigh Distribution

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Weibull Distribution  (shape parameter) allows reliability to increase or to decrease makes it equal to the exponential distribution (t)  =   t  -1 This is a useful model for trying to fit lots of different data sets, because it allows both increases and decreases.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Rayleigh Distribution: Same as Weibull with  = 2 (t)  =  t  -2 Some researchers have found that this distribution fits certain software cases.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 IEEE Reliability Standards

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version The Real Need A way to manage reliability throughout the life cycle of the software  This is the basis for the IEEE software reliability standard: “IEEE STD Software Reliability Metrics” & IEEE Guide (also see: IEEE STD Software Quality Assurance)

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version IEEE Software Reliability Standard 39 Measures applicability (1) Dobbins, James H., “SW Reliability Management,” in Handbook of SW Quality Assurance, Chapter 19.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version IEEE Standard Reliability Measures  Following are available for each measure –Description of use –Identification & definition of primitives –How it is implemented –How results are to be integrated –Special considerations –Special training or experience required –Specific example –Summary of benefits –Experience history –Published References

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Factors Contributing to Selection of IEEE Measures  Ease of collection of primitive data  Relationship between results & reliability  Ease of interpretation of results  Usefulness of results in management of the aspect being measured  Need for measurements in each aspect of each life cycle phase  Ease of implementation  Cost of implementation

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Three Basic Phases Predict Reliability Estimate Reliability Measure Reliability Start Coding Release Software Requirements Design Code Test Maintain Support

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version 7.09 Some of the IEEE Measures Unless otherwise specified, all of these are measured during development to help you assess, predict, estimate and/or control defect levels in your software.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Fault Density Total Faults per 1000 Lines of Code Used to –Predict remaining faults –Assess testing sufficiency –Establish historical data You can also track this after releasing the software

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Data Used to Measure Fault Density i = failure number n i = number of faults per failure N T =  n i = total faults found n f = fault density

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Fault Density Formula or n f = N T / KSLOC n f =  n i / KSLOC

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Graph of Fault Density

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Use of Fault Density  You can use fault density to estimate how close you are to finding all of the faults, and thus make decisions about whether to release software to the next development phase (or to the customer)  After software release, you can continue to measure (to see how well you estimated defects remaining)

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Additional Data Used to Categorize Faults d i = date of failure S i = severity of failure CL i = class of failure C i = type of fault These can help you determine which faults to focus on You may measure density separately for different classes or types or severity levels

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Faults Detected (per time period)  This can give you some idea of how well you are finding defects and thus how many defects are left in the software

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Faults Detected Per Week

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Density (pre-release)  This is similar to fault density, but is usually normalized by the size of the software  It tells you how your software compares with other software, so you have an idea of whether yours meets your norms or expectations You can also track this after releasing the software

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Density Definitions (pre-release)  A defect is a problem found during an inspection of the design, requirements, code, etc. i = inspection number D i = total defects found in ith inspection n = number of inspections  Defect density is the total defects, normalized by the size of the software

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Density Equation (pre-release) n DD =  D i / KSLOC i=1

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Using Defect Density  To use defect density, you should track how many you find before you release the product to the next phase  Over time, you can determine typical behaviors and thus meet goals or predict defect levels

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Density Requirements Analysis

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Density Software Design

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Density Code and Unit Test

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Requirements Traceability Goals: –Identify missing requirements –Identify features that are not required  Also known as “gold plating” –Measure progress in design and coding phases

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Requirements Traceability Equation RT = R1 / R2 * 100% Data needed to compute this: R1 = number of requirements traceable to specific design or code elements R2 = total number of requirements

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Requirements Traceability Variants and Usage Additional variants: –Trace to test cases –Trace from code/design/tests to requirements Usage notes: –This type of measure can be hard to keep up with and of limited utility if requirements change frequently. It works best if requirements are stable and defined.

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Defect Index (for each phase) Goal: –Provide an estimate of the relative correctness of the software Primitive Data: i = phase number N i = no of defects found in phase i Further Refinement: S i = # of serious defects found in phase i M i = # of medium defects found in phase i T i = # of trivial defects found in phase i

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Weighting for Defect Importance W S (typically 10) = weighting factor for serious defects W M (typically 3) = weighting factor for medium defects W T (typically 1) = weighting factor for trivial defects

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Phase Defect Index W S * S i W M * M i W T * T i Pi =______ + ______ + ______ N i N i N i

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Use of Phase Defect Index  This indicates the defect level of each phase  It can be used to see if a given phase is generating too many defects and, if so, whether they are important ones

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Overall Defect Index  i * P i DI =______ KSLOC

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Software Maturity Index Goal: –Help determine if we are ready for delivery Primitive Data: M T = # of functions in current release F c = # of functions changed since prior release F a = # of functions added since prior release F d = # of functions deleted from prior release

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Software Maturity Measure No 1: –For the most recently delivered software, count F c –For the next release, count F a, F d, M T M T - (F a + F c + F d ) MI =_______________ M T

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Software Maturity Measure No 2: M T - F c SMI =_______ M T

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Notes on IEEE Measures  These are indicators, not absolute measures  They must be calibrated to your data in order to be of most use  Despite the fact that they are over twenty years old, not very many practicing software organizations have used them –Cost –Not invented here –Fear of measurement

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version Summary  Simple models such as “seed and test” can give some concept of reliability  More complex distribution models may give better results if they match the behavior of the specific type of software  IEEE measures represent one set of possible metrics for estimating and managing reliability

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version References  IEEE STD 982.1, Software Reliability Metrics & IEEE Guide New York, Institute of Electrical and Electronics Engineers, Inc.  Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS ISBN

Copyright , Dennis J. Frailey CSE Software Measurement and Quality Engineering CSE8314 M00 - Version END OF MODULE 13