1. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 33 Quantitative Process Management

2. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 2 Outline Basic principles of QPM Statistical process control techniques Setting and using thresholds

3. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 3 Quantitative Process Management Typical Symptoms of a Problem We knew we were behind, but we didn’t know it was that bad! Problems just seemed to creep up on us until they overwhelmed us

4. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 4 Quantitative Process Management Symptoms of Another Problem The support costs are huge! Why did we ship that software with so many bugs in it? We never expected there would be so many when we released it.

5. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 5 Quantitative Process Management Other Characteristics & Symptoms You make decisions without factual data to justify them – Usually because you have not collected the right data – Or else you have not analyzed your data effectively Decisions may be inappropriate or harmful.

6. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 6 Quantitative Process Management Other Characteristics & Symptoms (continued) You overreact to normal variations in performance Or you allow things to get out of hand before taking action You don’t know the difference between a warning sign and a harmless variation

7. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 7 Quantitative Process Management Other Characteristics & Symptoms (continued) You can determine where the problems are but cannot assess which ones are the most serious You may solve the unimportant problems and overlook the ones that matter.

8. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 8 Quantitative Process Management Purpose To control the performance of the software project quantitatively. (Software process performance represents the actual results achieved from following a software process.)

9. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 9 Quantitative Process Management Definition Establishing quantitative goals for the performance of the project’s defined software process, based on knowledge of the capability of that process; Taking measurements of the process performance; Analyzing these measurements; and Making adjustments to maintain process performance within acceptable limits.

10. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 10 Why Do QPM? To enhance your ability to make informed decisions about what to do and how to prioritize your actions To set goals based on fact rather than opinion To give you the facts to determine whether changes actually improve the process To make you more competitive

11. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 11 How Do You Get There? 1) Characterize the process – Establish a capability baseline 2) Stabilize and manage the process – Improve consistency in the baseline so you have an expected range of values 3) Improve the process to achieve goals – Achieve a desired range of values

12. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 12 How it is Done in Practice Step 1 - Characterize Step 2 - Stabilize and manage Step 3 - Improve to meet goals Each step is ongoing once started, but you should do them in order to achieve the most effective results.

13. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 13 Step 1 Characterize the Process 1) Observe behavior 2) Establish a capability baseline – Average behavior – Variations in behavior 3) Measure actual behavior 4) Revise baseline as needed 5) Repeat until the baseline represents normal behavior Step 1 is a matter of measuring, observing, and revising baselines until they represent normal behavior

14. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 14 Typical Characterization Variation Range

15. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 15 Warning Too often, organizations set limits based on what they want instead of what they are capable of doing. An essential concept for the first step is to characterize your actual capability. Then, in future steps, you can find ways to improve so you can achieve what you want.

16. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 16 Step 2 Stabilize the Process Observe variance over time Seek sources of variance and improve the process to reduce them Over time, variance is reduced and baselines become stable Once behaviors are reasonably consistent, the process is said to be stabilized and you can establish an expected range of values

17. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 17 Types of Variations Special Cause variations - a specific project or project has a problem that causes it to vary substantially from the norm General Cause variations - problems inherent in the organization, the culture, or the process that are not specific to any particular project

18. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 18 Typical Un-stabilized Process Note that average moves significantly, and individual projects have wild swings.

19. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 19 Typical Stabilizing Process Average is more consistent, and individual projects have less severe swings. Expected Range

20. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 20 With a Stable Process You Can... Characterize expected behavior – Averages – Variations Establish an expected range of values Identify significant deviations from normal behavior – These are signs of a problem that must be addressed, normally a “special cause” problem

21. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 21 Step 3 Improve to Meet Goals 1) Observe capability of stabilizing process 2) Establish capability goals – Average behavior – Variations of behavior – I.e., a desired range of values 3) Identify ways to improve behavior 4) Move capability toward the goal 5) Repeat until the behavior achieves the goals

22. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 22 Quantitative Process Management Goals from the SEI CMM 1) The quantitative process management activities are planned – Planning occurs before the project starts 2) The performance of each project’s defined software process is controlled quantitatively … continued

23. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 23 Quantitative Process Management Goals from the SEI CMM 3) The process capability of the organization’s standard software process is known in quantitative terms

24. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 24 What do we Mean by Process Capability (Goal 3)? Capability is the range of values normally achieved by our process – Including a mean or average – And an acceptable level of variation from that mean In other words, we know what we are capable of and how we normally perform - expected range of values.

25. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 25 What do we Mean by Controlled Performance (Goal 2)? We can measure process performance against a capability standard We can determine when actual performance is outside of an accepted range We can correct performance to get it back within range

26. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 26 Typical Graph of Process Capability

27. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 27 Some Variability is Normal Performance between limits represents normal variability.

28. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 28 What is Normal Variability? “Too Perfect” Driver - No Variation“Typical” Driver - Normal Variation “Dangerous” Driver - Too Much Variation

29. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 29 Causes of Variability People – They vary somewhat from day to day Methods – They may not always work equally well on different applications Machines – They may have alignment variations, etc. Material – May vary from batch to batch Environment – Changes in the weather, for example

30. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 30 Knowing Capability means... Knowing what you can do Knowing what you cannot do Knowing what degree of variation is normal and what is not

31. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 31 Typical Graph of a Project vs Capability - Month 4

32. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 32 Typical Graph of a Project vs Capability - Month 4 Starting off in good shape

33. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 33 Typical Graph of a Project vs Capability - Month 6

34. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 34 Typical Graph of a Project vs Capability - Month 6 Upward trend suggests a potential problem

35. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 35 Typical Graph of a Project vs Capability - Month 8

36. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 36 Typical Graph of a Project vs Capability - Month 8 Crossing the limit shows a definite problem

37. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 37 Because We Know our Capability We Can Spot Problems Early We avoid the tendency to think things are only a little worse than last time Or to overreact to normal variation We can see our problems when we have time to react and take action We can more readily justify actions because the data show the trends and the risks

38. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 38 See the trend. Don’t wait for the crisis.

39. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 39 Note A process whose capability is known is not necessarily a good process – It may not meet acceptable performance or quality or cycle time standards – Performance may be highly unstable – It may be inefficient or poorly designed But you know what it is capable of in quantitative terms – So you know if you are performing at the level permitted by your process

40. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 40 References Benno, Stephen A., Dennis J. Frailey "Software Process Improvement in DSEG -- 1989 -1995" Texas Instruments Technical Journal, March- April 1995. Hudec, et. al., 1996. Experiences in Implementing Quantitative Process Management, Proceedings of SEPG Conference (SEI/IEEE).

41. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M33 8/20/2001Slide 41 END OF MODULE 33