CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 1 SMU CSE 8314 / NTU SE 762-N Software Measurement and Quality Engineering Module 14 Software Reliability Models - Part 2

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 2 Contents Requirements Volatility RADC Model Summary

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 3 Requirements Volatility [NOT an IEEE Metric, but Similar] Goal: Determine the stability of the requirements, so you can decide: – How far you really are in your development, – How reliable your software is likely to be, and – What type of process to use for software development

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 4 Requirements Volatility General Rules of Thumb If requirements are stable, use “waterfall” or similar processes If requirements are unstable, use incremental or evolutionary development

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 5 Requirements Volatility Primitive Data Collected R = # of original requirements – In original specification, for example C = # of changes to original requirements A = # of additions to original requirements D = # of deletions from original requirements

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 6 Requirements Volatility Equation V = (C + A + D) / R Very large V means unstable requirements You measure periodically to see if things are stabilizing

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 7 Typical Graph of Volatility

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 8 Requirements Volatility Usage Notes - 1 In a mature development effort for a production product, V should flatten out in the design phase, indicating stabilization of requirements If it continues to rise, it means you have an unstable development and should not be proceeding to later phases yet, unless this is a prototype effort

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 9 Requirements Volatility Usage Notes - 2 If V is large, the implication is that the current software development effort is really a requirements definition effort, which suggests a prototype, incremental or evolutionary development approach – If intended to be final development, do not go on to next step of process yet

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 10 Requirements Volatility Variation T = Number of “TBD” (“to be determined”) requirements in original specification This gives you more insight on changes that MUST happen (TBDs) It also gives more insight on stability over time V = (C + A + D + T) / R

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 11 Typical Graphs of V

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 12 You Can Also Learn a Lot by Graphing the Individual Factors of the Equation R = # of original requirements – In original specification, for example C = # of changes to original requirements A = # of additions to original requirements D = # of deletions from original requirements T = # of TBDs

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 13 Requirements Volatility Factors Sample Graph R V T C A D

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 14 Thresholds and Targets The nature of the development determines what thresholds should be established In a supposedly stable development, thresholds for stability should be very low - instability indicates development effort may be being wasted -- lots of rework ahead Continued...

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 15 Thresholds and Targets In a development that is expected to be volatile, thresholds might be high and targets would be established to determine when stability has been achieved. Historical data is essential for establishing reliable thresholds and targets

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 16 RADC Measurements Rome Air Development Center US Air Force Rome Air Force Base

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 17 RADC Measurements These are based on a large amount of data collected from U.S.A.F. Projects: – 5 million lines of code – 59 projects – Dating back to reliability models were studied Used as the basis for several government standards

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 18 Like IEEE, These Measurements Break the Process into Phases Predict Reliability Estimate Reliability Start Coding Release Software Requirements Design Code Test

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 19 Background of RADC Measurements Assumptions: – # of faults is fixed at the start of formal test – # of faults correlates to # of failures (failures are easier to measure and are the things the customer cares about) Goals: – Get the number of faults as low as possible – Predict number of faults as early as possible – Use Predictions to Manage the Process

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 20 Basic Approach to RADC Measurements for Reliability (one variant) Each factor that influences reliability is expressed as a number N 0 < N < 1 N = reliability impact of the individual factor N near 0 means it lowers reliability N near 1 means higher reliability The product of all these factors is the net reliability Each factor may be defined as the product of other, more detailed factors

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 21 RADC Concept R F1F1 F3F3 F2F2 F 21 F22F22 F23F23 R = F 1 * F 2 * F 3 F 2 = F 21 * F 22 * F 23 Assumptions: Factors are Bayesian, Independent, Homogeneous

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 22 Use of RADC Formula - I At start of project, you compute R and use it as the “current reliability prediction” As you go through the project, you try to improve the factors represented by the F i s, thus improving the value of R

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 23 Use of RADC Formula - II e.g. if F 3 represents programmer capability and it has a value of 0.6, you could improve it to 0.7 or 0.8 by hiring more capable programmers or by training your staff in defect reduction techniques Eventually, you base your values on actual results rather than on predictions

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 24 Reliability Expectation Improves Throughout the Lifecycle Estimates, based on Actual Code (R E ) Predictions, based on Factors Known at This Time (R P ) Goal (based on Specific System

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 25 Note: The Whole Thing Can Also Be Done in Terms of Other Factors Mean time between failures Probability of failure Hazard function Defects, etc. Regardless of how it is expressed, the idea is to: – Set goals based on system requirements – Determine the indicators for reliability – Improve early to achieve desired goals

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 26 What Factors does RADC Recommend? RADC has studied many software development efforts and has developed a recommended set of factors to use

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 27 Predicted Reliability R P = A * D * S A, D and S are factors known before you start developing the software

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 28 Predicted Reliability Factors A = Application type – Similar to Cocomo estimation model – Worse for embedded, real time, etc. D = development environment – Tools, turnaround, etc – Personnel capability also included S = software development methods & process – Factors included for each phase

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 29 S = Software Characteristics S = S 1 * S 2

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 30 S = Software Characteristics S 1 = Requirements & design methods & process – Structured Analysis, OO, etc. score higher – Less Formal techniques score lower – Process Management is a Big Factor S 2 = Implementation methods & process – Language – Coding standards – etc.

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 31 S 1 = Requirements and Design Methods S 1 = SA * ST * SQ

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 32 S 1 = Requirements and Design Methods SA = Anomaly management »Corrective action process »Risk tracking and contingency »etc. ST = Traceability »Ability to trace design to requirements, etc. SQ = Results of quality reviews »Number of defects found

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 33 SQ = Results of Quality Reviews Design Design Inspection Design Repair Design Inspection 27 defects SQ =.6 (too low) 3 defects SQ =.9 (OK) to Next Phase Note: values of SQ shown are illustrations. Actual values depend on size of code, defect definitions.

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 34 General Algorithm Do Something S i Too Low? Go On No Yes Redo

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 35 S 2 = Software Implementation Methods and Process S 2 = SL * SS * SM * SU * SX * SR

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 36 S 2 = Software Implementation Methods and Process SL = Language type – Higher order languages are better – Ada better than C due to Discipline, etc. SS = Program size SM = Modularity SU = Extent of reuse SX = McCabe complexity (of Design) SR = Review results (Defects Detected)

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 37 Examples of Improving S 2 Reduce design complexity Reduce number of defects allowed before exiting a review or inspection Use a better language Reuse proven code Make software more modular

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 38 Estimation Measurements (Based on Actual Code) F = Failure rate during testing T = Test environment (for Software) E = Operational environment During software test: R E = F * T During system test: R E = F * E

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 39 Diagram of Estimation Measurements Predictive Measureme nts R E = F * TR E = F * E

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 40 T = Test Environment TE = Test Effort -- amount of work spent testing TM = Test Methods -- sophistication, etc. TC = Test Coverage -- percent of paths tested, etc. T = TE * TM * TC

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 41 E = Operating Environment EW = Workload EV = Input Variability R E = F * E = F * EW * EV

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 42 Summary of RADC Measurement Usage - At Start of Project 1) Establish a reliability goal based on the objectives of the product 2) Using organization’s history, characterize the process in terms of the correlation between expected defect or reliability level and the various RADC parameters 3) Use this information to predict the reliability or defect level and use that information to affect the planning process – e.g. use to decide what language, CASE tools, etc.

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 43 Summary of RADC Measurement Usage - During Project Execution 4) Track actuals and compare with historical data and plans 5) Adjust behavior if actuals are inadequate to meet goals 6) Record actuals for future use

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 44 Summary Requirements volatility is easy to measure early in the project and it can give you a useful prediction of reliability and stability RADC and IEEE reliability models use different techniques for different phases of development RADC uses facts about the development process to predict reliability

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 45 References Bowen, C., et al., Methodology for Software Reliability Prediction, RADC-TR , Vol I & II, Rome Air Development Center, Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS ISBN Musa, John, Software Reliability Engineering: More Reliable Software, Faster Development and Testing, McGraw Hill. ISBN: Xie, M. Software Reliability Modeling, World Scientific, London, ISBN

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 46 END OF MODULE 14