1. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 1 SMU CSE 8314 / NTU SE 762-N Software Measurement and Quality Engineering Module 24 Product, Process and Project Measures

2. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 2 Three Different Types of Measures Product measures help us improve the product or similar products Process measures help us understand and improve the process Project measures help us understand and manage the project These are often based on the same data, but they differ in terms of what you are looking for and how you analyze the data

3. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 3 Product Measures Measures for the product are needed to help us assess and improve the product or similar products Examples: – Failures or defects by module – Test coverage (what percent of the product or the requirements do the tests cover) – Performance of product – Return ratio (defective products) – Design changes For more information, see Grady, chapter 6.

4. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 4 Example of a Product Measure for Performance Specification: must process 24 transactions per second Goal: process at least 24 transactions per second Metric: processing performance Measure: transactions processed per second This seems obvious, but how many organizations actually define/collect/monitor all of the key product performance measures?

5. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 5 Example of a Product Measure for Quality Goal: minimize failures in released products Definitions: – A failure is a neglect to match requirements – A defect is the cause of a failure We may measure failures or we might choose to measure defects

6. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 6 A Product Measure for Quality Sometimes it helps to ask a question. For example: Question 1: how likely are we to have failures after release? Metric: defect rate Measure: track # of defects throughout the lifecycle (at code reviews, etc.) – Keep track of how many are found, how many are fixed, where they were caused, etc. – Use this information to predict failures after release.

7. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 7 Measuring Defect Rate by Tracking Defects Each week you collect the following data: –Incoming Defects -- Defects newly detected this week –Corrected Defects -- Defects corrected this week Each week you compute the following measure: –Net Defects = Net (last week) + Incoming - Corrected

8. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 8 Measure 1 - Graph Planned Release Date Release Threshold

9. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 9 How Do We Set the Threshold? We must have historical data Collect two data points for each product: – defects known at release – total failures found after release (perhaps cut off after 12 months) Over time, we gain insight into the relationship between these two

10. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 10 Typical Scatter Chart of the Two Data Values for Each of Ten Products

11. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 11 Another Question Related to the Same Goal Goal: minimize failures in released products Question 2: where are the defects coming from and how can we reduce them? Approach: use defect data to identify causes and fix them.

12. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 12 Measure 2 -- Track Defect Containment This requires that you collect additional information about each defect: –In what phase of development was the defect created? –In what phase was it detected? This is really both a process measure and a product measure

13. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 13 In-Phase Defects In-phase defects are those that are corrected in the same phase where they were introduced - Example: a coding error that is caught and corrected before going to system test In-phase defects indicate which parts of your process generate the most defects In-phase defects are generally the least costly to correct.

14. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 14 Out-of-Phase Defects Out-of-phase defects are those that are detected (and corrected) after they leave the phase where they were introduced - Example: a design error caught during test Out-of-phase defects indicate how often you allow defects to “escape” the phase where they originate -- this is a good predictor of post-release failures and also a good help in root cause analysis Out-of-phase defects are generally the most costly to correct.

15. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 15 Step 1 -- Track Defects and Record Phase of Origin Defect Report Phase where found ____ Phase where introduced ___ ________________________ Priority _____ Type _____ Estimated Cost to Fix _____ etc.

16. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 16 Phase where Defect was Inserted Phase where Defect was Detected I&T C&T DDPD DD PD RA POST REL. 15 23 1783 5512 228 15 Step 2 - At the End of Each Phase, Record Phase of Origin

17. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 17 Using the Data If you see many out-of-phase defects in a specific cell, you can narrow down the source of defects Over time, you can correlate the number of defects in the matrix to the number of failures found by the customer, and can use this to predict and ultimately to manage the number of failures

18. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 18 Issues with This Method Definition of a defect must be adhered to in a consistent way across the project As shown, there is no distinction by type or severity of defect (but this distinction can also be made if the original data are good enough)

19. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 19 Key Lesson Learned from These Product Quality Measures If you detect and correct defects early, it greatly reduces cost and reduces post-release failures (i.e., those seen by the customer) - But this requires very good understanding of requirements and of customer “care-abouts”

20. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 20 Reluctance to Collect Defect Data There may be strong reluctance to collect defect data during development - The most professional software engineers develop an appreciation for the value of this type of information

21. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 21 Other Issues with Product Measures

22. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 22 Reliance on Hardware Measures Software is not bound by the laws of physics, and is not always bound by the laws of mathematics or hardware constraints. – Be careful not to rely on hardware measures where the theory assumes limits to physical behavior – Example: increase input by.0001%, output changes by 10000000000%. Rarely possible in hardware, but very easy in software.

23. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 23 Most Software Artifacts are NOT Code specifications tests user guides etc. If you only worry about defects in code, you will not learn all you could about your software and how to improve your process.

24. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 24 Software Products from Requirements Analysis Phase Data Dictionary Data Flow, Control Flow, or Entity Relationship Diagrams Requirements Specifications Interface Specifications Trace Matrices Test Requirements Plans Most of these are usually provided as documents!

25. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 25 Software Products from Design Phase Structure Charts Pseudo Code Test Designs Test Cases Control Charts etc. Again, mostly documents and intangibles. Yet Software Engineers often resist having quality measured in terms of defects in these things

26. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 26 Software Products from Code and Unit Test Phase Source Code Object Code Test Code Test Results Test Reports User Manuals Even here we have a lot of non-code products

27. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 27 Other Software Products Specifications Configuration Management Rules, Procedures, Directives, Naming Conventions, etc. “Make files” (loader control files) Programmer’s Reference Manuals Operator’s Reference Manuals...

28. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 28 Some Product Measures are Needed by Management Goals: – Assess Progress – Make Status Visible – Ensure Success Examples of things to measure: – Performance of product – Quality of interim products – Code complexity and conformance to standards

29. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 29 Code Complexity Measures Number of requirements met by each module or unit -- ideally small Number of units or modules for each requirement - ideally 1 Module size - ideally 50-100 LOC or less Interface simplicity Overall software complexity (number of modules and number of interfaces) Note potential tradeoffs -- none of these should be an absolute measure

30. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 30 McCabe Complexity (1) Correlates Strongly to Difficulty of Maintaining Code Correlates Weakly to Number of Defects in Code Graph of Complexity (produced by tool) helps software engineer understand code better and often leads to improvements Note: interface complexity and system design complexity as well as module complexity (1) McCabe, Thomas, IEEE Transactions on Software Engineering, December, 1976.

31. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 31 McCabe Complexity Measures for Code (1) Cyclomatic Complexity [of code] – Measures Control Flow -- Goal is 10 or less per module Essential Complexity [of code] – Measures Structure of Module - Goal is no greater than 2 – Measure AFTER making changes to fix defects -- can easily be affected by ill-conceived “patches” (1) See Grady, p88 and related pages.

32. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 32 McCabe Complexity Measures for Design and Integration (1) Module Design Complexity [can be measured on detailed design] – Measures the number of interfaces and effective use of subroutines Overall Design – Measures overall design approach and integration difficulty Integration Complexity – Measures testing effort (number of integration tests required)

33. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 33 Other Complexity Measures Fanout (1,2) – Correlates well with number of defects Design Weight (also from McCabe) (3) – Similar to Design Complexity, but based on data or control flow rather than detailed design or code – Can be used during preliminary design (1) Card & Glass, Measuring Software Design Quality, Prentice- Hall, 1990, p55. (2) see Grady, p80 (figure 7-12) (3) see Grady, p89, for more information

34. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 34 Testing Measures Branch Code Analysis – How well do the tests cover all branches – (see Grady, p90, Fig 8-5) Example (from Grady) Procedure# of Times Called# of PathsPaths Tested% X300,000 99100.0 Y02100.0 Z 7342882.4 Messages: Procedure Y is never tested Procedure X is over-tested

35. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 35 Note that This does NOT Assure that All Conditions are Tested IF A < 10 OR B < 20 THEN Q := SQRT(A) + SQRT(B) ELSE Q := SQRT(SQRT(A)) + SQRT(SQRT(B)) ENDIF Case 1: A = 5, B = 30 -- Tests the first path Case 2: A = 20, B = 30 -- Tests the second path BUT neither case tests for A=-3, B=-2!

36. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 36 Things NOT to Measure: Individual productivity or performance - This will cause people to generate invalid numbers - And it can easily penalize the most valuable contributors -- the ones who work on the hardest parts of the software Instead, put the focus on the process - Measure the product and use the information to improve the process - Measure people by their mature and effective use of process to improve overall results

37. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 37 Automate the Collection Process Software engineering tools (environments) can collect information. Advantages include: - Consistency- Integrity of Data - Low Cost for Collection- Ease of Use Be aware of metric collection possibilities when selecting tools such as design tools, debuggers, compilers, etc - CASE (Computer Aided Software Engineering) tools are excellent places to collect data But don’t overdo it -- you don’t need much

38. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 38 Configuration Management Tools are Especially Well Suited for Data Collection Items are submitted to CM at points when data are suitable -- e.g., at release to others: – specifications - code - test cases Much of the data is needed by CM anyway: -- e.g., line counts, defect counts, etc.

39. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 39 Getting People to Use CM The hard part is getting people to use CM early in the development process – May be viewed as unnecessary bureaucracy The other hard part is getting CM systems to be efficient when used during development

40. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 40 Trouble Report or Defect Report Systems - Good Sources of Data Standardization of definitions is desirable – What is a defect? – What are the priorities of defects? – Defect types – Defect causes – Origination points (what process caused the defect)

41. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 41 Challenges with Trouble Report or Defect Report Systems The hard part is getting people to document defects – Defect detection must be treated as a positive (getting defects out before ship date) rather than a negative (a sign that we made a mistake) – Mistakes must be recognized as process problems, not people problems

42. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 42 Process Measures These are often derived from product Measures They are used to help understand the process For example, analysis of defect containment data for all projects over a period of time or all projects using the same design methods may show such process information as: - Most frequent types of defects - Most costly defects - Time required to fix defects - Process steps generating the most defects - Which design standards help or hurt defects

43. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 43 Other Process Measures Trends in anything (process changes can change trends) Cycle Time for processes or parts of processes Productivity of processes Impact of process changes Inputs and Outputs of Process Steps – Quality (conformance to expectations/requirements) – On-time availability Cost of Process Steps

44. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 44 Example of Process Measures Defect Data from SA/SD Projects Defect Data from OO Projects SA/SD Defect PatternOO Defect Pattern

45. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 45 Another Example of Process Measures -- Add Historical Data to Product Defect Measure Planned Release Date Release Threshold

46. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 46 Issues with Process Measures Sometimes, you need to have consistent definitions across multiple projects It takes longer to collect enough data to generate meaningful results Changes in methods and processes must be understood so data are not misinterpreted - For example, data applicable to a structured design process may not accurately predict the results of an object oriented design process - Data from one type of application may not fit another

47. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 47 Project Measures These help you manage the project They tend to be focused on costs and schedules relative to plans or deadlines They tend to measure people They should measure things that effect productivity of people, such as: – Training – Turnover (planned and unplanned) – Resource utilization – Resource availability – Staffing level

48. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 48 Project Measures Quantify the Results of the Process They measure what the process is supposed to do If problems are uncovered, the solutions should be based on fixing the process Too often, the solutions only fix the immediate problem Example: Customer: “this soup is too salty” Waiter: “I’ll get you something else, sir” Customer: “but it is always too salty” Waiter: “I said I’d get you something else!

49. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 49 Sample Project Measures: TodayDeadline

50. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 50 Issues with Project Measures Established patterns are very hard to change - Use knowledge of patterns to manage - Expect difficulty changing patterns Adding resources introduces temporary delays - New staff learn while current staff teach them while work sits idle - New equipment must be installed, adjusted, etc.

51. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 51 Summary Measures can be used to understand the product, the process, and/or the project Make sure to measure important information about each of these – Too often, projects measure the project too much and the process too little – As a result they understand that they have problems but don’t understand why they have problems

52. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M24 version 3.09Slide 52 END OF MODULE 24