CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE 8314 / NTU SE 762-N Software Measurement and Quality Engineering Module 31 Methods of Observation Part 2 - Significance and Response (Displaying, Analyzing and Interpreting Data)

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 2 IntakeMeaningSignificanceResponse (*) Satir, Virginia et al. (references) The Observation Process (*) Previous Module This Module Other Modules

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 3 Methods of Display and Interpretation (*) (*) Weinberg, Vol 2, chapters 4-8; Grady, chapters 2, 12 (references) Significance

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 4 Methods of Display and Interpretation “Diagrams are Nothing; Diagramming is Everything” -- Weinberg, after Eisenhower (*) (*) Eisenhower, Dwight, “Plans are Nothing, Planning is Everything.”

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 5 Methods of Diagramming The way you present the data is associated with the message you want to convey It is important to know what you want to communicate before selecting a way of graphing the data An important objective of many types of diagrams is to show the relationships between different factors

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 6 Methods to be Described Fishbone Diagrams Scatter Charts Histograms Pareto Charts Run (Trend) Charts Control Charts

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 7 Fishbone (Ishikawa) Diagrams Project Late Resources Design Workstations Skills Motivation Errors People Complexity See previous module for more discussion

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 8 Scatter Charts Purpose: – To show correlation (or lack of correlation) between two variables Method: – Plot two or more variables on an x-y or scatter chart

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 9 Scatter Chart Example

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 10 Positive Correlation V2 = a * V1 + b

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 11 Negative Correlation

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 12 No Apparent Correlation

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 13 Further Categorization of Data Sometimes the scatter chart hides a relationship because we did not segregate the data sufficiently well – e.g, suppose the last diagram showed fault density and weeks late on shipping – And suppose we had two kinds of projects: those that do inspections and walkthroughs and those that use only testing to identify defects – If we further segregate the data, we might see the following

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 14 Segregated Scatter Chart

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 15 Notes about Correlation Correlation does NOT necessarily mean cause and effect Correlation can come in many shapes

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 16 Histogram Purpose: – To show the variance in a collection of data – Usually the data are expected to cluster about a mean Method: – Use a bar or column chart to show data values as a function of some variable – Can also show frequency of occurrence vs. data value

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 17 Histogram Example Second Data Value vs. Core Index

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 18 Histogram Example Unexpected Variation

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 19 Histogram Example # of Occurrences vs. Core Index

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 20 Low Variability about a Mean

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 21 High Variability about a Mean

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 22 Skewed Data Target Value

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 23 Uncontrolled Data Target Value

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 24 Do Projects with Walkthroughs Ship On Time More Often?

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 25 No, but they have Fewer Defects

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 26 Pareto Charts Purpose: – To identify the most significant cases – To highlight where to focus – To separate the significant few from the trivial many Method: – Sort data by vertical axis (“y”) value

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 27 Example Days Lost due to Vacation

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 28 Example No Apparent Discriminator Showing only the top of each bar helps show differences Sometimes, Pareto analysis shows that there is no significant difference

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 29 Discussion The previous charts are good at showing relationships between factors But they do not say much about the meaning of an observation – What does it mean to say that there are 5 defects per KLOC in the output? – Is this good, bad, typical, ??? For such purposes we need to show reference points in our data

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 30 Reference Point Example Corporat e Goal

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 31 Run Charts or Trend Charts Purpose: – To show the variance over time – To show how much variation is normal – To help understand what constitutes normal variance and what constitutes exceptional data Method: – Plot all data using a line chart and then compute and (optionally) plot the average as a separate line Note that the average is based on current data, not past history

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 32 Run Chart Example Data relative to Recent History

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 33 Run Chart Example

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 34 Run Chart with Moving Average

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 35 Control Charts Purpose: – To track performance – To know when a process or a machine is performing out of its normal range – To know when to take action Method: – Show actual data vs. average and expected variation (control limits). – Very much like a run chart, but with control limits added and with average based on prior data rather than current data

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 36 Typical Control Chart

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 37 More on Control Charts This will be addressed in a later module on quantitative process management

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 38 Recommended Reading Weinberg, vol. 2, Chapter 5 -- Slip Charts (see references)

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 39 Data Analysis Significance

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 40 Data Analysis Improper analysis can lead to wrong conclusions Proper analysis is very hard, it requires: – Insight into the problem – Knowledge about software development – Knowledge about the application – Knowledge of the customer situation – Tracking down the real facts – Looking at the data in several ways Telling the difference can be even harder!

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 41 Example of Need for Proper Analysis

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 42 Naive Conclusion Don’t inspect designs or code. Wait until code is done because it is cheaper to find and fix the defects while testing.

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 43 Proper Analysis Shows... Each phase detects different defects – Those introduced early may not be detected during code and test phase PhaseType XType YType Z RA PD DD C&T I&T

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 44 Proper Analysis Also Shows... Net cost for post-release defects is higher for those introduced in early stages Defect Correction Cost by Phase when Defect was Introduced

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 45 An Alternative Way to Evaluate Inspections and Walkthroughs Track the net gain for inspections and walkthroughs D = defects detected per inspection T = time (staff hours) per inspection f = time (staff hours) to fix a defect after an inspection F = time (staff hours) to find and fix defects after release – F R = F for requirements defects – F D = F for design defects – etc. (i.e., subscript is phase of origin)

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 46 Tracking the Gain Metric D * (F-f) - T = Gain for the inspection

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 47 Analyzing the Impact of Defect Prevention Activities Collect key post-release data – Incoming defects – Repair time & cost per defect (staff hours) – Calendar time per defect – Phase during which defect was introduced Compare different projects to see impact

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 48 Project 1 Phase% of DefectsAvg Fix IntroducedCost Req Des Cod Test Total Project 2 Phase% of DefectsAvg Fix IntroducedCost Req Des Cod Test Total Comparison of Two Projects Project 1 invested more money in up front activities and ended up with a significantly lower net cost to fix defects. We also need to understand the total number of defects.

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 49 What About Taking Action? This module is primarily about interpreting things properly Other modules address various ways of taking action on the basis of measurements Response

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 50 Summary Drawing proper significance from data depends on methods of displaying the data and interpreting the results Proper analysis avoids incorrect conclusions

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 51 References Satir, Virginia et al., The Satir Model, Family Therapy and Beyond, Palo Alto, CA., Science and Behavior Books, ISBN: Weinberg, Gerald M. Quality Software Management, Volume 2, First Order Measurement, Dorset House, New York, ISBN:

CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 52 END OF MODULE 31