1. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 29 Measuring and Improving the Software Development Process

2. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 2 Outline The Metrics Process Tying Metrics to the Software Development Process Root Cause Analysis

3. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 3 The Metrics Process

4. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 Metrics Process Manage Risks Select Collect Analyze Utilize

5. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 5 The Metrics Process Define goals – Start with a customer – Prioritize and align Define metrics – Work with affected parties Define questions and measurements – See prior module on selecting SW metrics Define the data to be collected – Focus on clear definitions Select

6. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 6 The Metrics Process Plan and define the data collection process – Consider the need for culture change – Consider the impact of data collection Communicate the process to affected parties Deploy the collection process Collect the data Record and store the data Collect

7. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 7 The Metrics Process Validate the Data – Errors are common – Often due to misunderstandings Report the Data – Display – Interpret Analyze the Data – Establish organizational knowledge Analyze

8. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 8 The Metrics Process Change the project – Identify trends early – Adjust plans to head off problems Change the process – At the project level – At the organizational level Change the metrics process – Data definitions, collection process, validation, etc. Utilize

9. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 9 Each Measurement Must Have an Objective and a Customer Objective must be clear – Otherwise, people will not know why and may resist or may collect the wrong thing Interpretation of the data must be clear – Otherwise, metrics can be misused and misinterpreted End user (customer) of the measurement must want the information – Otherwise, all of the effort is for no purpose

10. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 10 The Objective can be Stated in terms of Three Elements To understand / evaluate / control / predict / report An attribute of an entity (resource, process, product) In order to satisfy a purpose (related to the goal of the metric)

11. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 11 Construct a Sentence for Each Measurement “We are measuring MMM so we can OOO the AAA of the EEE in order to PPP” – MMM is a measure – OOO is an objective – AAA is an attribute – EEE is an entity – PPP is a purpose “We are measuring the number of lines of code so we can understand the size of the software in order to predict the cost and schedule of the project.” This idea is due to Linda Westfall (see references)

12. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 12 Purpose and Use must be Communicated to All Concerned If people do not know why they are being measured, they will mistrust the measurements and generate bad data

13. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 13 Use of Metrics must be Demonstrated

14. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 14 Tying Metrics to the Software Development Process

15. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 Managers -- Project metrics – That’s what they are measured by – But if the project is in trouble they need to know more Developers -- Product metrics – That’s what they are measured by Both should care about Process metrics – This is usually where you learn the reasons for a problem Who Cares about What?

16. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 What Should we Measure? ProductProjectProcess determines success of determines quality of root causes Process Metrics – How effective is the process? – How well are we following the process? – Risk monitoring Product Metrics – Performance and quality – How well is the product meeting its requirements ? Project Metrics – Used to monitor the state of the project – How are we doing relative to cost, schedule, staffing, etc.?

17. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 Example: Same Data, Different Uses Data: Number of defects in the software Use in a Project Metric: number of defects must be less than target before project is complete, thus the data becomes a measure of project status. Use in a Product Metric: product quality index is “defects per 1000 LOC”. Used to determine probable warranty cost. Use in a Process Metric: quality index is compared for different processes and for process improvements to determine which processes are best for future projects in the organization

18. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 18 How do you Fix Problems Identified by Project Metrics? An effective solution is usually based on analyzing and fixing the process

19. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 Example: A Measure & its Impact Measure 1: Lines of code per day Use: reward those who produce the most lines of code per day Result: people produce bloated, inflated code in order to look good Measure 2: Requirements met and tested, weighted by complexity of requirement Use: track against history and use to identify process bottlenecks Result: people will use the data to make the process more efficient, resulting in lower cost

20. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 Good and Not-So-Good Measures Goal: Produce software more efficiently Metric: Efficiency Measure 1: tests completed per week Result: easy tests done first; corners cut in testing; hard problems ignored or deferred Measure 2: rework Result: process and methods are improved to reduce rework, resulting in more efficient software development But rework is a lagging indicator - it does not spot problems in advance

21. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 ProductProjectProcess Attributes What Resources Quality Time Are We On Schedule? Expenses vs. Budget? How Fast can we Manufacture? What Is our Cycle Time? Post-release Defects? What will it Cost? What is our Productivity? Customer Satisfaction? In-process Defects? Performance Meets Perf. Goals? Meets Mgt. Goals? Does it Work? What Attributes can we Measure? We want attributes that relate to our goals – time, resources, performance, quality etc. The following type of matrix can help:

22. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 Tie Metrics to the Process -- A Measure for Every Phase Phase Metric Require ments Test Plan ning Design Integr ation Coding Staffing X X XXX X Requirements Stability XXX Etc. X Design Complexity X Code Complexity

23. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 23 Plan the Metrics Process to be Part of the Software Process The software process and associated procedures define the day-to-day actions – Thus they are the ideal place to communicate details of how metrics should be collected & evaluated The software process is defined in terms of tasks to be done, inputs and outputs, and sequencing

24. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 24 Typical Process Description Design Softwar e Module Inspect Design Softwar e Design Review Place Module under Configurat ion Control Design Module Test Cases Place Tests under Configurati on Control defects

25. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 25 “Inspect Design” Detailed Process Description Hold Inspection Meeting Track Defects to Closure Record Defects Detected Identify Inspection Participan ts Decide if OK to Release to CM Design Software Module Software Design Review Place Module under Configuration Control Design Module Test Cases Inspect Design Place Tests under Configuration Control defects

26. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 26 “Record Defects Detected” (text description of task) For each defect, identify the following: – Type – Origin – Severity – Date and Time Found – Person assigned to Correct – Estimated effort to Correct Record the above in the defect database When defect is corrected, record date and time corrected and total effort to correct

27. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 27 Tie the Software Process to the Metrics Consider metrics for each process task or procedure or input or output – Why (Goal of this measurement) – What to collect (primitive data; clear definitions) – How to collect (forms, procedures, automation) – Responsibility for collection (individuals, organizations) – How it will be used (analysis, formulas, etc.)

28. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 28 Tie the Software Process to the Metrics (continued) Add this information to the process description -- usually as part of the text description of the bottom level task But don’t overdo it And make sure there is a concrete benefit for each measurement being recommended.

29. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 29 Process for Design Walkthrough 1) Collect design documents 2) … (walkthrough details)... 3) … (walkthrough details)... 4) Identify Defects 5) Categorize by Priority and Type 6) Document in Walkthrough Report 7) Define action plan for Each Defect 8) Assign each defect to Analyst for Resolution 9) Report status each week until closure, using “open defects” report Another Example focusing on Status Metrics

30. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 30 Typical Graph of Status Metric Open Defects Report for module XXX123

31. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 31 Root Cause Analysis

32. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 32 Root Cause Analysis Primarily used with defect metrics Used to identify causes of defects so you can change the process to prevent them The methods can also be used to trace forward: – given a defect, what problems might it cause Unfortunately, the terminology differs greatly among various authors.

33. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 33 IEEE Standard for Defect Classification See IEEE standards for latest status Rather elaborate and comprehensive Tends to be rather generic and therefore hard to apply to specific projects This illustrates one of the problems with standards -- they are often too universal and generic to be of much use for specific applications

34. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 34 Hewlett-Packard Model for Defect Analysis Each defect is classified in terms of three factors: 1) Origin (phase of process where defect originated) – Specification/Requirements – Design – Coding – Environment and Support – Documentation – Operator or Other See Grady pp 127- 129, and Leath (references)

35. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 35 Typical Results from Origins Analysis

36. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 36 Origins Analysis Weighted by Cost to Fix Defects (from HP) Weighting Factors : Specification14.25 Design 6.25 Code 2.50 Documentation 1.00 Operator 1.00 Other 1.00

37. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 37 Hewlett-Packard Model (continued) 2) Type of Defect (what caused the problem) – Specification error – Functionality (impossible or incorrectly described) – Interface (hardware, software, or user; design or implementation) – Interprocess communication – Data definition or data handling error – Module design

38. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 38 More Types of defects More types of defects – Logic error or description error – Error checking – Standards specified or applied incorrectly – Computation error – Test error (hardware, software, process) – Integration error – Tools error

39. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 39 Distribution of Defects by Type (from HP)

40. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 40 Hewlett-Packard Model (continued) 3) Mode (why it happened) – Something missing – Something unclear – Something wrong – Something changed – A better way is known

41. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 41 Advanced Uses Organizations with more experience and data may identify defects in very complex ways, such as: – Most expensive defect types – Most time consuming defect types – Most customer sensitive defect types – Most frequent defect types

42. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 42 Changing the Process They may also identify process changes that will reduce the number of defects Examples: – Change the measure by which the project manager is rewarded – Eliminate wasteful documentation – Plan testing earlier in the process

43. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 43 Ishikawa Diagrams Useful for brainstorming possible causes of problems The idea is to categorize defects and then subcategorize them to try to reach a better understanding of what kinds of defects are occurring and why Basic approach is a “fishbone” picture where the main “backbone” is the category of defect and the “ribs” are subcategories Grady & Caswell, p. 127; Ishikawa (see references)

44. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 44 Concept Category of Defect Subcategory Cause

45. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 45 Example Register Allocation Error Incorrect Usage Side Effect of Correct Usage Incorrect Documentation Lack of Knowledge Keep Track Improperly Poor Design Documentation

46. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 46 Beware of Linear Thinking Just because Joe is always associated with register allocation defects, does not mean that Joe is the cause of these defects Perhaps Joe is the person who wrote the code that produces the problem, but that does not mean Joe is the problem Perhaps the problem is inherent in the requirements, for example

47. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 47 Many Authors have Categorized Defects in Various Ways Materials, Workers, Tools and Inspections Materials, Methods, Measures Process, Operator, Equipment and Material Process, People, Tools, Inputs FURPS+, etc. Each “rib” on the fishbone diagram might be one of the defect categories.

48. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 48 Example People Tools Inputs Process

49. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 49 An Interesting Finding from HP Experience using Ishikawa Over half of the errors reported occurred during redesigns These were traced to the fact that design reviews were not carried out during redesigns because they occurred in a hurried environment Consider the design flow diagram of a few pages back -- did we re-inspect the corrections?

50. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 50 Summary The metrics process has four steps: – Select – Collect – Analyze – Utilize Construct a sentence to make sure that each measurement has an appropriate objective

51. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 51 Summary (continued) Measure project, product and process – Measure project and product to identify problems – Measure process to identify causes – Use process information to identify organizational characteristics and trends Tie measurements to the software development process

52. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 52 Summary (concluded) Determine Causes of Problems – Root cause analysis – Ishikawa diagrams Beware of Linear Thinking

53. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 53 References Grady, Robert B. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, N.J., Prentice-Hall, Inc., 1992. ISBN 0-13 ‑ 720384-5. Grady, Robert B. and Deborah L. Caswell, Software Metrics: Establishing a Company-Wide Program. Englewood Cliffs, N.J., Prentice-Hall, Inc., 1987. ISBN 0-13-821844-7. Ishikawa, K. Guide to Quality Control, Asian Productivity Organization, Tokyo (1976), 18-28.

54. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 54 References (continued) IEEE, A Standard for Software Errors, Faults and Failures, IEEE Working Group P1044, March, 1987. Leath, C., “A Software Defect Analysis,” HP Software Productivity Conference Proceedings, (April, 1987) 4:147-161 Westfall, Linda, Software Metrics that Meet Your Information Needs, Dallas SPIN Tutorial, October 11, 1994.

55. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M29 8/20/2001Slide 55 END OF MODULE 29