1. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 1 SMU CSE 8314 / NTU SE 762-N Software Measurement and Quality Engineering Module 23 Using Measurements Effectively

2. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 2 Contents More on Terminology Graphing and Analyzing Data

3. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 3 More on Terminology

4. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 4 Metrics, Measures and Data These words are often used interchangeably But their meanings are different And as practiced in software engineering, they often have distinct and specific meanings The ISO standards discussed in the previous module attempt to clarify this

5. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 5 Dictionary Definitions Metrics – The art or science of measurement – A standard of measurement Measure – The dimensions, capacity or amount of something Data – Factual information used as a basis for reasoning, discussion, measurement or calculation

6. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 6 So What Does it All Mean? Data are collected to measure (data are the basis of measurements) Measures are used to meet some information need (measures are the basis information)

7. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 7 The Relationship Entity Being Measured Base Measure Information Need Derived Measure

8. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 8 Example Process in Execution Units Produced Head- count Lines of Code $ SpentMonths Productivity Units Per Month LOC per Staff Month $ per Line of Code

9. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 9 An Information Need... “... An insight desired to understand something” Productivity – How efficient are we? Cycle Time – How fast do we execute the process? Cost – How many resources must we expend? etc. Note that an information need can usually be expressed as a question

10. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 10 Information May Be Obtained From Many Different Measures! Cost is best measured in staff days! Why not dollars ? I measure it by drop in stock price. However they measure it, their problem is how to reduce cost!

11. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 11 Key Issues for Selecting Measures A measure should provide insight or information about a Goal or Purpose An effective measure is usually associated with some higher level objective, such as “reducing cost” or “on time delivery” [This will be covered in more detail in a later module]

12. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 12 There are Typically Many Ways to Satisfy an Information Need You must spend time to select the right measure, so you achieve the higher level objective(s) We will address this more in the next few slides, as we discuss measurements

13. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 13 A Measurement Procedure is... “... a specific formula or graph or computation that provides a means of evaluating the the size or dimensions or capacity or other attributes of somethin g” – This definition focuses on the use of measurement to evaluate something

14. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 14 How Do You Know What Measures and Measurement Procedures to Use? Derived measures are generally chosen because they answer a question or address an information need To select an effective derived measure, you need to have a mental model of what is important

15. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 15 A Measure is... – A measure provides something we can analyze & interpret “the result of quantifying an attribute of a process or product” “the result of applying a measurement procedure”

16. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 16 Examples of Measures Productivity Measures – Lines of code per staff day, or – Units produced per year, or – Copies sold/month less returns/month Customer Satisfaction Measures – Sales per month – Customer complaints per product – Warranty costs per year

17. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 17 How Do You Choose? By knowing the information need And having a mental model of what will satisfy that need This is usually related to how you intend to analyze and interpret the data Which means you need to think about analysis and interpretation when selecting measures

18. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 18 Measures Allow Interpretation and Analysis Prediction Decision Alerts Status Evaluation Implications Assessment

19. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 19 Selecting a Measure is Not Always Easy How do you measure Cycle Time? – Calendar days between contract & delivery – Working days from contract to qualification test – Sum of cycle times of sub-processes – Average cycle time for all products – Weighted average cycle time for all products – Mean and variance of cycle time for all products Each of the above is a reasonable measure

20. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 20 One Way to Select a Measure... Understand the goal & the question that the measure should answer Goal: – Faster response to customer orders Question or Information Need: – How long does it usually take us to respond to an order? Measure: – Cycle time for the process, defined as calendar days between contract & delivery

21. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 21 Different Goal, Different Measure Goal: – To understand how efficiently we are executing the process. Question or Information Need: – How long does it usually take us to carry out the process? Measure: – Cycle time for the process, defined as working days between contract & delivery

22. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 22 Mini Assignment See if you can think of at least five ways to measure quality At least five ways to measure efficiency At least five ways to measure the value of a stock (how an investor might value a stock) Do not turn anything in. This is a thought experiment or a basis for group discussion.

23. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 23 Key Issues for Selecting Measures Proper interpretation of the measure should be defined to form a basis of consistent analysis Graphing technique is also important [We will discuss these in more detail in a later module on collecting and analyzing data]

24. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 24 How Will you Use the Data? People do not like to be measured Especially if they don’t know why they are being measured or how the data will be used So you must demonstrate that you are using it the way you claim you are, so people will believe you

25. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 25 Measurement Causes Change Given any measurement, people will change to make it the data work to their advantage So you want to make their behavior change in a positive way

26. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 26 Base Measures are... Examples – Number of employees – Days since some event – Units produced – Lines of code in a module – Defects reported Base measures are independent of each other … the actual attributes of the entities being measured

27. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 27 Key Issues in Defining Base Measures You must have a precise definition of what you want For example, if you are defining hours worked, you must define – How many hours are there per work day? – How do you count overtime?

28. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 28 How Long is a Work Day? Salaried Staff Work Day OvertimeRegular Time Hourly Staff Salaried Staff Hourly Staff UnpaidPaidUnpaid Paid The definition should be accurate and consistent. Which of these do you include?

29. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 29 Imprecise Definitions can lead to Useless Data How much overtime was used? – All time spent beyond 8 hours per day? – All time actually recorded on time sheets? – All time spent by managers or just software developers? – Contract labor personnel or just regular? – Subcontractor personnel or just your own?

30. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 30 Key Issues in Defining Base Measures (continued) Can you collect the data accurately ? Can you collect data consistently across multiple projects? Can you collect it efficiently? How will the organization & the process change when you collect the data?

31. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 31 “Domestic Violence Data Called Muddled” Dallas Morning News, January 1, 1996 COLUMBUS, Ohio - Every month, police officers around the country meticulously compile bundles of domestic-violence statistics... [but it is] a waste of time. That’s because many other police departments regularly fail to report the same kinds of calls and arrests, making statewide totals meaningless. And those who do make reports use assorted definitions for basic categories, making comparisons impossible

32. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 32 Observation As the organization gets more mature – It tends to value precise definitions of its measures – Models also tend to play a more prominent role in organizations to understand the process

33. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 33 Caution About Using Models But models are only models -- do not use them to predict things they are not capable of – Example: Cocomo cost estimating model predicts that you cannot develop software faster than a particular speed – But some people do it faster because they are using techniques not included in the model

34. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 34 What to Store in a Data Base Base Measures? Derived Measures? Results of Analysis?

35. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 35 Raw Data of Base Measures is the Best Thing to Maintain in a Historical Data Base It is less biased It has many different uses, some of which you don’t currently know It is typically used in many calculations for different measures But it can take up a lot of storage – Data reduction techniques can help More on data bases in a later course in this series

36. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 36 Graphing and Analyzing Data

37. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 37 Graphs Very often, a picture is much easier to interpret than raw data Original Plan Actual Hoped Likely Today

38. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 38 Graphs are Helpful for Interpreting Data Prediction - extrapolate trends Decision - look for signs of trouble Alerts - look for deviations from expectations Status - see plans vs. actuals Evaluation - compare alternatives Implications - project outcomes Assessment - evaluate issues

39. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 39 Graphs are Also Employed for Misusing Measurement! Option 3 appears to be Significantly Better

40. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 40 The Real Difference! In fact, all three are roughly equal!

41. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 41 A More Accurate Graph!

42. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 42 But if it were a race... The last graph would not show who won. It all depends on why you are looking at the data

43. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 43 Another way to Assess Data Raw data can be categorized This can give insight into the true facts

44. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 44 Report of Total Defects Tentative Conclusion: Defects are out of control

45. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 45 Report of Total Defects by Type Tentative Conclusions: – Important defects under control – Not enough manpower for minor ones

46. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 46 More on Categorization Categorizing defects or bugs requires skill and judgment Customer and engineer do not see it the same way Many issues are technically complex

47. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 47 More on Categorization Politics - Some ways of categorizing are politically incorrect – Defects vs years of experience Probably OK – Defects vs level of training Probably OK – Defects vs gender or race or religion or ethnic origins Often not OK Unless you are trying to prove bias

48. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 48 Summary Good measures relate to the goals and the questions (information needs) Derived measures are based on models of what is important Base measures must be precisely defined, factual and readily obtainable Graphs and categories help to interpret -- or misinterpret the data

49. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 49 As You Listen to the Remaining Modules in the Course Consider which ones are addressing which aspect of Measurement

50. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 50 References Department of Defense, Joint Logistics Commanders Joint Group on Systems Engineering, Practical Software Measurement, a Guide to Objective Program Insight (version 2.1), Naval Undersea Warfare Center, c/o John McGarry, mcgarry@ada.npt.navy.mil. Tufte, Edward, The Visual Display of Quantitative Information, Graphics Press, 1983

51. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M23 version 3.09Slide 51 END OF MODULE 23