1. Lero© 2010 Software Quality & Process Improvement Dr. Ita Richardson Lero – the Irish Software Engineering Research Centre and Department of Computer Science & Information Systems University of Limerick 1 Lero© 2010

2. LECTURE 3: MEASURING THE SOFTWARE PROCESS

3. Lero© 2010 Why Measure? Business & Industry use quantitative approach Measure: – Product quality – Productivity – Efficiency – Machine reliability Why should Software Engineering be different?

4. Lero© 2010 Quantitative Measures Length of project activity? Time remaining in project? Productivity of software engineer? Bottlenecks in system? % of time for test vs development No of errors detected Time spent fixing errors

5. Lero© 2010 Measurement as feedback Performance indicators Match against best practice Make required improvements Results in continuous process performance

6. Lero© 2010 Measurement Measurement in chaotic environment is generally not useful Need consistent and standard process Use balanced set of indicators Measure versus goals

7. Lero© 2010 Types of measures Process-related measures – Number of change requests – Cycle time – Effectiveness of the process Project-related measures – Project development time – Cost of work to date – Deviation of costs – Staff productivity

8. Lero© 2010 Types of measures Product / Customer –related measures – User-reported bugs – Customer satisfaction – Source code size

9. Lero© 2010 Principles of SP Measurement Define clear objectives – Key performance indicators Management support – Supporting Business Goals Roles and responsibilities Cover both process and product Reflect business & engineering objectives

10. Lero© 2010 Issues to be considered How to measure business value of SPI? – Increased productivity – Early error detection and correction – Cost of SPI How to measure qualitative benefits? – Product availability – Customer satisfaction – Employee morale

11. Lero© 2010 Measurement Tools Understanding Data – Histograms – Bar Charts – Pareto Charts – Scatter diagrams – Run charts Analyse Data – Control Charts

12. Lero© 2010 Using the Data How do we analyse the data? What can we do within the organisation? Do we need more information? Should cause us to ask questions as well as answer questions

13. Lero© 2010 Histograms Frequency counts Distribution of observed values Easy to compare distributions

14. Lero© 2010 Histogram

15. Lero© 2010 Bar Charts Sets of discrete values Data associated with individual entities

16. Lero© 2010 Bar Chart

17. Lero© 2010 Scatter Diagrams Comparing one variable with another ‘Dependent’ / ‘Independent’ Cause and Effect

18. Lero© 2010 Scatter Diagram

19. Lero© 2010 Run Charts Values arranged in time sequence Basis for Control Charts Temporal Behaviour of Process

20. Lero© 2010 –,, © S-Cube – 20/ Run Chart

21. Lero© 2010 Sequential Run Chart As Run Charts BUT some Grouping of Data exists

22. Lero© 2010 When did the training occur? Sequential Run Chart

23. Lero© 2010 Pareto Charts Ranks outcomes Frequency counts in descending order Temporal

24. Lero© 2010 Variation Total Variation = Common Cause Variation + Assignable Cause Variation

25. Lero© 2010 Common Cause Variation Natural Results are common across groups Stable consistent pattern Predictable Unexpected results are rare

26. Lero© 2010 Assignable Cause Variation Causes which could have been prevented Outside influences Shifts in quality – People – Processes – Tools

27. Lero© 2010 Stable process In statistical control No assignable causes – As they have been removed from the process Variability due to Common Causes Use Statistical Process Control – Control Charts

28. Lero© 2010 Control Charts How process has behaved R-Chart – Range charts – Smallest: Largest value – Don’t expect to exceed limits of range variation X-Chart – in control – Sub-group averages don’t fall outside upper/lower variation control For examples of Control Charts: see the internet

29. Lero© 2010 Statistical Process Control Improve process though causal analysis Control Limits – Estimated for process – Cannot assign arbitrarily – +- 3 sigma (Std Deviations) Centreline – Observed process average

30. Lero© 2010 Statistical Process Control Common Cause Variation – Assignable cause must be eliminated Rational Sub-Grouping & Rational Sampling – To eliminate common cause variation – To ensure narrow control limits

31. Lero© 2010 Out-of-control Tests Instability where one of the following exist: Single point outside 3-sigma Two / three successive values fall on the same side and outside 2-sigma Four / five successive values fall on the same side and outside 1-sigma Eight successive values fall on the same side of centreline – Tests 2-4 for X-Charts only

32. Lero© 2010 What do we do with Results? Remove assignable causes then Change the process then Continually improve

33. Lero© 2010 Remove assignable causes Find reasons for their existence Prevent their recurrence Introduces stability to process May use qualitative data May use other quantitative data

34. Lero© 2010 Change the process Removing assignable causes requires: Identifying changes Designing changes Implementing changes

35. Lero© 2010 Continually improve Reduce variability Improves quality Improves cost Improves time-to-market Ultimate requirement is STABLE & CAPABLE process - Need to consider Change Management

36. Lero© 2010 Framework for Change Management (Willman, 1996) Pressure for Change Leadership & Vision Capable People Effective Rewards= Successful Implementation Pressure for change Leadership & Vision Capable People ? = Evaporation Pressure for change Leadership & Vision Capable People Effective Rewards=Frustration Pressure for change Leadership & Vision ?Effective Rewards=Disengagement Pressure for change ?Capable People Effective Rewards=Disillusionment ?Leadership & Vision Capable People Effective Rewards=Disinterest

37. Lero© 2010 Measurement as feedback Performance indicators Match against best practice Make required improvements Results in continuous process performance

38. Lero© 2010 Acknowledgement The information presented in these slides has been collected from a variety of sources including: – Software Quality Assurance: From Theory to Implementation by Daniel Galin, 2003 – Software Process Improvement: Practical Guidelines for Business Success by Sami Zahran, 1998 – Research carried out by post-doctoral researchers and PhD students at Lero – the Irish Software Engineering Research Centre, Ireland under the supervision of Dr. Ita Richardson – Software Process: Improvement and Practice (journal) – The SPIRE Handbook: Better, Faster, Cheaper Software Development in Small Organisations, edited by Marty Sanders (Version 1, 1998) and Jill Pritchet (Version 2, 2000) The research presented in this lecture has been partially supported by Science Foundation Ireland funded through Global Software Development in SMEs Cluster Grant (no 03/IN3/1408C) and Lero – the Irish Software Engineering Research Centre (CSET grant no 03CE2/I303.1).