1. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 13 Software Reliability Models - Part 1

2. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 2 Contents Seeding and Tagging Weibull Distribution Rayleigh Distribution IEEE SW Reliability Standard Summary

3. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 3 Seeding and Tagging

4. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 4 Seeding and Tagging Introduce a given number of errors into the software -- say E of them Run standard tests, detecting D of them Compute D/E = % of errors detected Suppose D 2 = number of other errors detected Then you assume the total number of errors in the software is D 2 *E/D

5. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 5 Example of Seeding and Tagging 200 defects found so far Inject 20 defects Find 12 of them Therefore, assume total defects = 200 * 20 / 12 = 4000 / 12 = 333 => 333 - 200 = 133 defects remaining By performing this analysis from time to time, you can estimate your defect density over time.

6. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 6 Distributions Other than Exponential Weibull Distribution Rayleigh Distribution

7. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 7 Weibull Distribution  (shape parameter) allows reliability to increase  or to decrease   makes it equal to the exponential distribution (t)  =   t  -1 This is a useful model for trying to fit lots of different data sets, because it allows both increases and decreases.

8. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 8 Rayleigh Distribution: Same as Weibull with  = 2 (t)  =  t  -2 Some researchers have found that this distribution fits certain software cases.

9. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 9 IEEE Reliability Standards

10. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 10 The Real Need A way to manage reliability throughout the life cycle of the software This is the basis for the IEEE software reliability standard: “IEEE STD 982.1 Software Reliability Metrics” & IEEE Guide 982.2 (also see: IEEE STD 730-1984 Software Quality Assurance)

11. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 11 IEEE Software Reliability Standard 39 Measures applicability (1) Dobbins, James H., “SW Reliability Management,” in Handbook of SW Quality Assurance, Chapter 19.

12. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 12 IEEE Standard Metrics Following are available for each metric – Description of use – Identification & definition of primitives – How it is implemented – How results are to be integrated – Special considerations – Special training or experience required – Specific example – Summary of benefits – Experience history – Published References

13. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 13 Factors Contributing to Selection of IEEE Metrics Ease of collection of primitive data Relationship between results & reliability Ease of interpretation of results Usefulness of results in management of the aspect being measured Need for measurements in each aspect of each life cycle phase Ease of implementation Cost of implementation

14. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 14 Three Basic Phases Predict Reliability Estimate Reliability Measure Reliability Start Coding Release Software Requirements Design Code Test Maintain Support

15. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 15 Some of the IEEE Metrics Unless otherwise specified, all of these are measured during development to help you assess, predict, estimate and/or control defect levels in your software.

16. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 16 Fault Density Total Faults per 1000 Lines of Code Used to – Predict remaining faults – Assess testing sufficiency – Establish historical data You can also track this after releasing the software

17. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 17 Data Used to Measure Fault Density i = failure number n i = number of faults per failure ----- N T =  n i = total faults found n f = fault density

18. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 18 Fault Density Formula or n f = N T / KSLOC n f =  n i / KSLOC

19. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 19 Graph of Fault Density

20. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 20 Use of Fault Density You can use fault density to estimate how close you are to finding all of the faults, and thus make decisions about whether to release software to the next development phase (or to the customer) After software release, you can continue to measure (to see how well you estimated defects remaining)

21. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 21 Additional Data Used to Categorize Faults d i = date of failure S i = severity of failure CL i = class of failure C i = type of fault These can help you determine which faults to focus on You may measure density separately for different classes or types or severity levels

22. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 22 Faults Detected (per time period) This can give you some idea of how well you are finding defects and thus how many defects are left in the software

23. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 23 Faults Detected Per Week

24. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 24 Defect Density (pre-release) This is similar to fault density, but is usually normalized by the size of the software It tells you how your software compares with other software, so you have an idea of whether yours meets your norms or expectations You can also track this after releasing the software

25. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 25 Defect Density Definitions (pre-release) A defect is a problem found during an inspection of the design, requirements, code, etc. i = inspection number D i = total defects found in ith inspection n = number of inspections Defect density is the total defects, normalized by the size of the software

26. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 26 Defect Density Equation (pre-release) n DD =  D i / KSLOC i=1

27. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 27 Using Defect Density To use defect density, you should track how many you find before you release the product to the next phase Over time, you can determine typical behaviors and thus meet goals or predict defect levels

28. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 28 Defect Density Requirements Analysis

29. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 29 Defect Density Software Design

30. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 30 Defect Density Code and Unit Test

31. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 31 Requirements Traceability Goals: – Identify missing requirements – Identify features that are not required Also known as “gold plating” – Measure progress in design and coding phases

32. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 32 Requirements Traceability Equation RT = R1 / R2 * 100% Data needed to compute this: – R1 = number of requirements traceable to specific design or code elements – R2 = total number of requirements

33. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 33 Requirements Traceability Variants and Usage Additional variants: – Trace to test cases – Trace from code/design/tests to requirements Usage notes: – This type of measure can be hard to keep up with and of limited utility if requirements change frequently. It works best if requirements are stable and defined.

34. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 34 Defect Index (for each phase) Goal: – Provide an estimate of the relative correctness of the software Primitive Data: i = phase number N i = no of defects found in phase i Further Refinement: S i = # of serious defects found in phase i M i = # of medium defects found in phase i T i = # of trivial defects found in phase i

35. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 35 Weighting for Defect Importance W S (typically 10) = weighting factor for serious defects W M (typically 3) = weighting factor for medium defects W T (typically 1) = weighting factor for trivial defects

36. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 36 Phase Defect Index W S * S i W M * M i W T * T i Pi =______ +______ + ______ N i N i N i

37. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 37 Use of Phase Defect Index This indicates the defect level of each phase It can be used to see if a given phase is generating too many defects and, if so, whether they are important ones

38. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 38 Overall Defect Index  i * P i DI =______ KSLOC

39. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 39 Software Maturity Index Goal: – Help determine if we are ready for delivery Primitive Data: M T = # of functions in current release F c = # of functions changed since prior release F a = # of functions added since prior release F d = # of functions deleted from prior release

40. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 40 Software Maturity Measure No 1: – For the most recently delivered software, count F c – For the next release, count F a, F d, M T M T - (F a + F c + F d ) MI =_______________ M T

41. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 41 Software Maturity Measure No 2: M T - F c SMI =_______ M T

42. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 42 Notes on IEEE Metrics These are indicators, not absolute measures They must be calibrated to your data in order to be of most use Despite the fact that they are over ten years old, not very many practicing software organizations have used them – Cost – Not invented here – Fear of metrics

43. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 43 Summary Simple models such as “seed and test” can give some concept of reliability More complex distribution models may give better results if they match the behavior of the specific type of software IEEE metrics represent one set of possible metrics for estimating and managing reliability

44. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 44 References IEEE STD 982.1, Software Reliability Metrics & IEEE Guide 982.2. New York, Institute of Electrical and Electronics Engineers, Inc. Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS00030. ISBN 0-07-039400-8.

45. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M13 8/20/2001Slide 45 END OF MODULE 13