1. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 38 Rethinking the Management Process

2. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 2 Metrics are Easy Change is Hard Metrics often tell us that we need to change But we often resist change, despite the best of intentions Sometimes the problem is with the process by which we manage - because we are uncomfortable changing that

3. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 3 Typical Management Behavior Immature software process – Poor quality – Difficult to meet schedules – Grief for software manager Denial (hide or ignore data) Blame Tradeoffs (dealing) Decision to change Change [but target is moving!!!] Repeat the cycle

4. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 4 Two Basic Approaches for Dealing with Need to Change 1 Aggregate (Shotgun) –Try anything and hope 2 Feedback Control (Rifle) –Try well conceived things based on models, knowledge and observation

5. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 5 Aggregate Approach Initiate many projects and improvements The best survive (capitalism) – Costly, but it works – Useful for purchased software (try all and pick the best)

6. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 6 The Aggregate Approach Happens Often Sometimes you do this unintentionally – “Keep trying until you find something that works” – Parallel efforts with different initial goals or different methods But intentional use can occur at all maturity levels

7. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 7 Aggregate Approach at Maturity Level 1 At this level there is no management of change, just hope – Natural selection – Costs money – Takes a long time – Does eventually work, if you stick with it

8. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 8 Aggregate Approach at Maturity Level 2 Uneducated management / operates by rote – Controlled use of multiple approaches Often an effective and efficient approach – But may omit the best ones! No true understanding

9. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 9 Aggregate Approach at Maturity Level 3 Educated management – Carefully chosen approaches – Carefully motivated use – Reward for best solution This approach is more likely to spot and nurture really good solutions – But you may make the mistake of selecting by emotion rather than by fact

10. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 10 Aggregate Approach at Maturity Level 4 Quantitative understanding of processes – Explore alternatives selected on the basis of sound, quantitative data – Use well chosen data to evaluate alternatives At this level you are less likely to miss the best solution because of erroneous “gut feel”

11. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 11 Aggregate Approach at Maturity Level 5 Continuous process improvement – Similar to level 4, but incorporated into the overall organizational process – No special effort required because it is an ongoing part of the way you do business

12. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 12 Feedback Control Approach Requires more investment Less costly and more effective in the long run

13. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 13 Feedback Control Make a model of the system Sophistication of model depends on maturity level Manager modifies system based on inputs and behavior and what the model says about the impact of changes

14. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 14 Level 0 or 1 Model of a System resources random factors requirements System or Process software other artifacts At this level there is no external control Programmers want: –Unlimited resources –No changes (no random factors) –Fixed requirements

15. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 15 Level 2 Model of a System resources random factors requirements System or Process software other artifacts control At this level there is external control But still no access to the internals of the system or process

16. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 16 Examples of Level 2 Control Training : make them all smarter Tools : make them all seem smarter Hire MIT Grads : make some of them smarter Hire Harvard Grads : make some of them seem smarter See Weinberg

17. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 17 More Examples of Level 2 Control Pay them : motivate better performance Give them interesting assignments : motivate better performance Fire all Berkeley grads (nonconformists) : motivate the rest Note 1: you still have randomness Note 2: manager must control in order to feel that she/he is contributing to the effort

18. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 18 Management at Level 2 Failure to plan -- don’t know what to expect or what to do Planning by rote -- well established procedures and processes and metrics for specific parts of software development Failure to observe -- don’t know whether to do anything Failure to compare -- don’t know if product is right

19. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 19 Symptoms of Level 2 Management Reviews emphasize quantity: – Number of hours spent – Number of lines produced Rather than quality: – Number of defects – Number of requirements met Note that requirements met and defects are much more accurate estimates of the amount of work actually completed

20. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 20 Level 3 Model of a System At this level the manager can do more effective planning and can delegate control to the software engineers resources random factors requirements System or Process software other artifacts control knowledge of system and outputs

21. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 21 Management at Level 3 Good planning – you know what to expect and what to do Planning is guided by history / standards – well established procedures, processes and metrics are the starting point, not the end point Observation – becomes the main role of the software manager

22. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 22 What Management Knows at Level 3 Knowledge of the product Knowledge of the process as it is really happening

23. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 23 Requirements for Observation Visible evidence Stable process A difference between computer science and disciplined software engineering is in the ability to observe … … which means … an appreciation of the importance of a stable process and a knowledge of how to achieve it. Weinberg

24. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 24 Suggestions for Level 3 Management Change by addition, not deletion or criticism Control is not the same as low defect count – True control means knowing what defects are important and which ones should be permitted to stay Feedback goes from the process to the manager, not the other way

25. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 25 More Suggestions for Level 3 Management Be willing to revert to aggregate methods when prudent – Example: Disk crashes -- how to recover? – If caused randomly, there may not be an established procedure you need to try everything

26. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 26 Levels 4 and 5 Level 4: Feedback control is applied to the process as well as to the product Level 5: Feedback control is applied to the culture as well as to the process and product

27. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 27 Notes on System Models Consider a car... “Is that clicking sound significant?” -- Car owner may not know -- Or worse, may think she knows based on experience with older cars (new ones don’t work the same way) -- Mechanic has current knowledge Lack of knowledge is incompetence Lack of current knowledge can be unconscious incompetence

28. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 28 Linear Models Consider a Software Development Effort -- 5000 lines of code can be done in 32 staff months -- So 10000 lines of code will take 64 staff months, right? === The above ignores integration! Non-linearity is one of the chief reasons why software efforts get into trouble Scaling often does not work

29. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 29 Impact of New People Work Done Progress = --------------------------------- Total Work To be Done In order to increase progress, you want to increase the numerator and decrease the denominator But adding new people decreases the numerator and increases the denominator (although the numerator effect is temporary)

30. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 30 Diagram of Effects New People Training Load on Old Workers Additional management, overhead, etc Work Completed goes DOWN (temporarily) Work to be Done goes UP (permanently) Lower Productivity (permanent) Slower Progress (temporary)

31. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 31 The Value of Diagrams Drawing Diagrams is Very Helpful The drawing process gives insight The drawing helps communicate

32. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 32 Measurement Observations “You can design a measurement system for any conclusion you wish to draw.” -- Tom DeMarco Consider car advertisements comparing features The car being promoted always seems to come out best

33. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 33 Effective Use of DeMarco’s Principle Problem: Programmer A is good Programmer B is not so good Programmer A does not want to help B (hurts A’s performance)

34. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 34 Solution Effect: A teams with B Both do better in the long run New metric for rewards: A is told that her duties as a more senior technical contributor include mentoring the less experienced people, such as B Both A and B will be rewarded based on the average performance of A and B

35. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 35 Gene Miluk’s Questions Part 1 - Direct Impact For each measure you want to make, ask the following questions: – Why am I measuring it? – How will I measure it? – What is the desired outcome of measuring it?

36. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 36 Gene Miluk’s Questions Part 2 - Indirect Impact For each measure you want to make, ask: – What is a possible undesirable outcome? – What behaviors must be avoided to avoid such outcomes? – What is a possible balance impacted? Example: quality vs. cost and schedule – How can I keep the balance proper?

37. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 37 Summary The software manager must move from being the orchestrator of chaos -- to The controller of order -- to An observer of order

38. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 38 Observation by Jim Coplien (AT&T) “The most efficient and effective software projects are those where the communication between management and workers is relatively low in volume and well controlled.”

39. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 39 Summary Management must change as an organization matures, so as to make most effective use of metrics Both aggregate (shotgun) and feedback control (rifle) approaches are useful, although the latter is often more effective in the long run A mature organization is characterized by more management observation and less feedback

40. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M38 8/20/2001Slide 40 END OF MODULE 38