CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE.

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE 8314 / NTU SE 762-N Software Measurement and Quality Engineering Module 31 Methods of Observation Part 2 - Significance and Response (Displaying, Analyzing and Interpreting Data)

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 2 IntakeMeaningSignificanceResponse (*) Satir, Virginia et al. (references) The Observation Process (*) Previous Module This Module Other Modules

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 3 Methods of Display and Interpretation (*) (*) Weinberg, Vol 2, chapters 4-8; Grady, chapters 2, 12 (references) Significance

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 4 Methods of Display and Interpretation “Diagrams are Nothing; Diagramming is Everything” -- Weinberg, after Eisenhower (*) (*) Eisenhower, Dwight, “Plans are Nothing, Planning is Everything.”

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 5 Methods of Diagramming The way you present the data is associated with the message you want to convey It is important to know what you want to communicate before selecting a way of graphing the data An important objective of many types of diagrams is to show the relationships between different factors

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 6 Methods to be Described Fishbone Diagrams Scatter Charts Histograms Pareto Charts Run (Trend) Charts Control Charts

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 7 Fishbone (Ishikawa) Diagrams Project Late Resources Design Workstations Skills Motivation Errors People Complexity See previous module for more discussion

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 8 Scatter Charts Purpose: – To show correlation (or lack of correlation) between two variables Method: – Plot two or more variables on an x-y or scatter chart

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 13 Further Categorization of Data Sometimes the scatter chart hides a relationship because we did not segregate the data sufficiently well – e.g, suppose the last diagram showed fault density and weeks late on shipping – And suppose we had two kinds of projects: those that do inspections and walkthroughs and those that use only testing to identify defects – If we further segregate the data, we might see the following

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 15 Notes about Correlation Correlation does NOT necessarily mean cause and effect Correlation can come in many shapes

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 16 Histogram Purpose: – To show the variance in a collection of data – Usually the data are expected to cluster about a mean Method: – Use a bar or column chart to show data values as a function of some variable – Can also show frequency of occurrence vs. data value

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 26 Pareto Charts Purpose: – To identify the most significant cases – To highlight where to focus – To separate the significant few from the trivial many Method: – Sort data by vertical axis (“y”) value

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 28 Example No Apparent Discriminator Showing only the top of each bar helps show differences Sometimes, Pareto analysis shows that there is no significant difference

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 29 Discussion The previous charts are good at showing relationships between factors But they do not say much about the meaning of an observation – What does it mean to say that there are 5 defects per KLOC in the output? – Is this good, bad, typical, ??? For such purposes we need to show reference points in our data

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 31 Run Charts or Trend Charts Purpose: – To show the variance over time – To show how much variation is normal – To help understand what constitutes normal variance and what constitutes exceptional data Method: – Plot all data using a line chart and then compute and (optionally) plot the average as a separate line Note that the average is based on current data, not past history

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 35 Control Charts Purpose: – To track performance – To know when a process or a machine is performing out of its normal range – To know when to take action Method: – Show actual data vs. average and expected variation (control limits). – Very much like a run chart, but with control limits added and with average based on prior data rather than current data

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 37 More on Control Charts This will be addressed in a later module on quantitative process management

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 38 Recommended Reading Weinberg, vol. 2, Chapter 5 -- Slip Charts (see references)

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 40 Data Analysis Improper analysis can lead to wrong conclusions Proper analysis is very hard, it requires: – Insight into the problem – Knowledge about software development – Knowledge about the application – Knowledge of the customer situation – Tracking down the real facts – Looking at the data in several ways Telling the difference can be even harder!

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 42 Naive Conclusion Don’t inspect designs or code. Wait until code is done because it is cheaper to find and fix the defects while testing.

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 43 Proper Analysis Shows... Each phase detects different defects – Those introduced early may not be detected during code and test phase PhaseType XType YType Z RA 80 20 10 PD 40 30 30 DD 30 40 50 C&T 10 40 80 I&T 5 30 45

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 44 Proper Analysis Also Shows... Net cost for post-release defects is higher for those introduced in early stages Defect Correction Cost by Phase when Defect was Introduced

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 45 An Alternative Way to Evaluate Inspections and Walkthroughs Track the net gain for inspections and walkthroughs D = defects detected per inspection T = time (staff hours) per inspection f = time (staff hours) to fix a defect after an inspection F = time (staff hours) to find and fix defects after release – F R = F for requirements defects – F D = F for design defects – etc. (i.e., subscript is phase of origin)

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 46 Tracking the Gain Metric D * (F-f) - T = Gain for the inspection

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 47 Analyzing the Impact of Defect Prevention Activities Collect key post-release data – Incoming defects – Repair time & cost per defect (staff hours) – Calendar time per defect – Phase during which defect was introduced Compare different projects to see impact

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 48 Project 1 Phase% of DefectsAvg Fix IntroducedCost -------------------------------------- Req. 2600 Des. 20300 Cod. 75 80 Test 5 46 ---------------------------------- Total100125 Project 2 Phase% of DefectsAvg Fix IntroducedCost -------------------------------------- Req. 5575 Des. 15235 Cod. 65118 Test 15 75 ------------------------------------ Total100142 Comparison of Two Projects Project 1 invested more money in up front activities and ended up with a significantly lower net cost to fix defects. We also need to understand the total number of defects.

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 49 What About Taking Action? This module is primarily about interpreting things properly Other modules address various ways of taking action on the basis of measurements Response

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 50 Summary Drawing proper significance from data depends on methods of displaying the data and interpreting the results Proper analysis avoids incorrect conclusions

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 51 References Satir, Virginia et al., The Satir Model, Family Therapy and Beyond, Palo Alto, CA., Science and Behavior Books, 1991. ISBN: 0831400781. Weinberg, Gerald M. Quality Software Management, Volume 2, First Order Measurement, Dorset House, New York, 1993. ISBN: 0-932633-24-2.

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE.

Similar presentations

Presentation on theme: "CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE.

Similar presentations

Presentation on theme: "CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M31 version 5.09Slide 1 SMU CSE."— Presentation transcript:

Similar presentations

About project

Feedback