January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 1 CHAPTER 10 MEASURING AND MONITORING NOTICE: This material is copyrighted and may be copied or downloaded ONCE ONLY by students who are registered in this course at Southern Methodist University or National Technological University.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 2 Outline 10.1 Overview 10.2 Basic Issues - Levels of Metrics and Attributes Measured 10.3 Selecting Metrics 10.4 Recommended Metrics 10.5 Metrics Issues

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # Overview Where metrics fit in the larger planning, tracking and oversight picture Basic reasons for measurement Possible results - good and bad

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 4 Metrics are used to Monitor Monitoring is used to Manage Risk Manage Risks Define the Approach Generate Detailed Plans Understand the Need Monitor Execution text, chapters 6, 15

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 5 SW Project Monitoring Typical Symptoms of a Problem We’re six months behind schedule, and nobody knew it! Why did it take us so long to find out? January

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 6 SW Project Monitoring Symptoms of Another Problem They’ve been working on that module for eight weeks and everyone else is waiting for it. Did the developers in charge know how many people depend on the module?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 7 SW Project Monitoring Other Examples The project manager promised a new feature to the customer -- but never told any of the programmers! – “I thought you knew about this!” The software takes up too much disk space. – Nobody ever thought it would get that big.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 8 SW Project Tracking and Oversight Purpose To provide adequate visibility into actual progress so that management can take effective actions when the software project’s performance deviates significantly from the software plans

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 9 SW Tracking and Oversight Goals from the SEI CMM 1) Actual results and performance are tracked against software plans – Plans are revised to reflect actual performance and changes in requirements or commitments 2) Corrective actions are taken and managed to closure when actual performance deviates significantly from software plans

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 10 3PM Today SW Tracking and Oversight Goals from the SEI CMM 3) Changes to software commitments are communicated to all affected groups and individuals

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 11 SW Project Tracking & Oversight Practices Recommended by SEI CMM Use a software development plan for tracking and communicating status Track schedule, size, effort, computer resources, technical activities and risks Hold periodic reviews and take corrective actions Revise plans and schedules to reflect changes -- using a defined procedure Review customer commitments on a regular basis

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 12 Things you can Estimate and Monitor Costs Sizes Quality Reliability Schedules Staffing etc.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 13 Establish a Data Base Know about your organization – Performance on past projects – Lessons Learned Know about your industry and competitors – What is best in class? – Improvement rates Historical Data Base - Metrics - Lessons - etc. Facts to help you manage

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 14 Example of Experience vs. History History: for C++ doing your kind of software, you should be generating – 25 lines of code per day during the coding phase, with – 3 errors per 1000 lines of code during module test Actual experience on your project: – 40 lines of code per day, with – 0.5 errors per 1000 lines of code during module test

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 15 Optimist’s Conclusion Do you have a solid reason to explain this difference? Ask questions. Why are you better? – Is the process different? – Are the people a lot better? – Are the tools better? We are doing much better than in the past!

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 16 Pessimist’s Conclusion Ask questions. Find out what is really happening. – Are the tests being performed? – Is the coverage adequate? – Are there higher rates of customer complaints after shipment? Our testing is no good (perhaps because it is being rushed due to deadlines)

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 17 Knowing the Competition Can Give You Insights But what if the norm in your industry is an improvement of 25%? And what if your competitors have all switched to Java and are 50% more productive as a result? We improved our “C” language productivity by 15%

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 18 Every Metric should have a Purpose -- You want to get Information Data Analysis Information

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 19 But for Every Analysis there are Two Possible Results Information - tells you something right – We are (or are not) on schedule – Our risks are (or are not) under control Misinformation - tells you something wrong – We are (or are not) on schedule – Our risks are (or are not) under control And there will always be changes in the organization when you measure it

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 20 Key Issues Define how to interpret measurements – To form a basis of consistent analysis Choose consistent graphing techniques – So people know how to interpret the data Define how to use each measurement – You must also demonstrate that you are using it that way, so people will believe you – Given any measure, people will change to make it look to their advantage you want to make their behavior change in a positive way

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 21 Organizational Framework There must be an Organizational Framework for understanding the importance of metrics – i.e., people do not sabotage the metrics collection effort – And people do not abuse metrics – And people do not draw wrong conclusions

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 22 Achieving an Effective Organizational Framework Educate everyone in the proper use of measurements Develop the right metrics – Involve those who are being measured – Measure only what you can benefit from Use the measurements – To make decisions about the product and the process – But NOT to make decisions about people

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 23 You Want to Achieve Optimal Performance Don’t over-measure or under-measure Don’t over-test or under-test Don’t over-inspect or under-inspect etc. Track the things that represent your greatest risks and concerns Remember that it costs time and money to track - make it worthwhile

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # Basic Issues Levels of Metrics What to Measure Who Cares about What Metrics

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # : Levels of Metrics What do we mean by a metric? Does everyone mean the same thing? How do we resolve discrepancies? Consider the example on the next page:

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 26 Metrics may be assessed or measured in many ways! Cost is best measured in staff days! Why not dollars ? I measure it by drop in stock price. However they measure it, they had better figure out how to reduce cost!

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 27 Metrics, Measures and Data Process in Execution Data Metric Measure

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 28 Example Process in Execution Units Produced Head- count Lines of Code $ SpentMonths Productivity Units Per Month LOC per Staff Month $ per Line of Code

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 29 Definitions: Data Data (or Primitive Measures) The fundamental, factual quantities that characterize a process or product – Specific Facts that are countable or otherwise obtainable – Minimum amount of Analysis

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 30 Definitions: Data A Key Issue is Consistent Definitions Another Key Issue is Accuracy of Collection Examples: – Hours worked – Number of employees – Lines of code

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 31 Data (or Primitive Measures) Factual Information with Minimal Interpretation -- Examples: days, lines of code, people, dollars, defects, variance, tests completed, complaints -- You must define what you want -- How many hours per day? -- Which lines to include in LOC?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 32 Data (or Primitive Measures) Factual Information with Minimal Interpretation -- Can you collect accurate and consistent data? -- Can you collect it efficiently? -- How will the organization and the process change when you collect this data?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 33 How Long is a Staff Day? Salaried Staff Staff Day OvertimeRegular Time Hourly StaffSalaried StaffHourly Staff UnpaidPaidUnpaidPaid

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 34 Definitions: Measures Measure (sometimes called Compound Measure) The result of counting or otherwise quantifying an attribute of a process or product – How we quantify – Something we can analyze and interpret

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 35 Definitions: Measures A Key Issue is Proper Interpretation How Measures are Graphed is Also Important Examples: – lines of code per hour – ratio of current to historical test performance – turnover rate

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 36 Measures (or Compound Measures) Measures are values computed from data or other measures -- Generally associated with specific graphs and rules of interpretation -- Examples: lines of code per day; defects per thousand lines of code; return on investment

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 37 Measures (or Compound Measures) You must define how you will use it You must also demonstrate that you are using it that way, so people will believe you Given any measure, people will change to make it look to their advantage -- so you want to make their behavior change in a positive way

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 38 A Typical Graph of a Measure

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 39 Example: A Measure & Its Impact Metric: Productivity Measure 1: Lines of code per day Use: reward those who produce the most lines of code per day Result: people may produce bloated, inflated code in order to look good Measure 2: requirements met and tested, weighted by complexity of requirement Use: track against history and use to identify process bottlenecks Result: people may use the data to make the process more efficient, resulting in lower cost

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 40 Definitions: Metrics Metric A measure or combination of measures that provides insight into a software issue or concept. – What we Measure – An Indicator – A Standard of Measurement that ties to a Goal or Purpose

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 41 Definitions: Metrics A key issue is selecting a measure that provides insight or information r.e. the goal or purpose Examples: – Comparisons (planned vs. actual) – Indices (performance ratio, productivity, return on investment)

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 42 Metrics Groups or Categories or Systems of Measures Focus on the Goals -- Examples: productivity, quality, cycle time, defect density, customer satisfaction -- Each metric is usually associated with some higher level objective, such as “reducing cost” or “on time delivery”

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 43 Metrics Groups or Categories or Systems of Measures Focus on the Goals -- There are many ways to measure a metric and different organizations measure differently -- The key is to spend time to select the right measure, so you achieve the higher level objective(s)

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 44 Good and Not So Good Measures Goal: Produce software more efficiently Metric: Efficiency Measure 1: tests completed per week Result: easy tests done first; corners cut in testing; hard problems ignored or deferred Measure 2: rework Result: process and methods are improved to reduce rework, resulting in more efficient software development But rework is a lagging indicator - it does not spot problems in advance

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # What Should we Measure? ProductProjectProcess determines success of determines quality of root causes Process Metrics – Effectiveness of the process – How well are we following the process? – Risk monitoring Product Metrics – Performance and quality – How well is the product meeting its requirements? Project Metrics – The state of the project – How are we doing relative to cost, schedule, staffing, etc.?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 46 ProductProjectProcess Attributes What Resources Quality Time Are We On Schedule? Expenses vs. Budget? How Fast can we Manufacture? What Is our Cycle Time? Post-release Defects? What will it Cost? What is our Productivity? Customer Satisfaction? In-process Defects? Performance Meets Perf. Goals? Meets Mgt. Goals? Does it Work? What Attributes can we Measure? We want attributes that relate to our goals – time, resources, performance, quality etc. The following type of matrix can help:

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # : Who Cares about What? Managers usually care about project metrics - that’s what they are measured by -- But if the project is in trouble they need to know more Developers usually care about product metrics -- that’s what they are measured by Both should care about process metrics -- this is usually where you learn the reasons why a project or a product is in trouble

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # Selecting Metrics Goal/Question/Metric Mapping to the Process Tying to Risks

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 49 Selecting Metrics: The Goal- Question-Metric Paradigm (1) GOAL (what do we want to accomplish) QUESTIONS (the answers tell us whether we are meeting the goals or help us understand how to meet them) METRICS (measurements that answer the questions) (1) Victor Basili & David Weiss, “A Methodology for Collecting Valid Software Engineering Data,” IEEE Transactions on Software Engineering, Vol SE-10, no. 6, Nov., 1984, p728.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 50 Steps of the GQM Paradigm 1Determine your Goals -- and Prioritize Them – Don’t Select Too Many – Make Sure they are your Real Goals 2For each Goal, Determine What Questions would Help you Understand Whether you are Achieving the Goal or How to Achieve it

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 51 Steps of the GQM Paradigm 3For each Question, Determine if it can be Answered by Measurement – If so, determine what can be measured – And what it would cost to measure – And the side effects of the measurement 4Select the Metrics with the Best Payoff

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 52 Remember that Measures are Not Goals If the measure becomes the goal, you usually end up at the wrong place (example: lines of code per day) The measure tells you how you are doing, but there may be other measures that tell you other aspects of how you are doing You Get What You Measure!

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 53 Prioritize Goals You cannot always measure everything Determine which goals are most important Also make sure that the goals are consistent Are people getting consistent direction?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 54 It Helps to Have a Vision Vision Key Features or Goals Prioritized List of Goals Focus Metrics on Top Priority Goals 1. Learn to Sail 2. Select the Best Beer 3. Save $ 1. Save $ 2. Learn to Sail

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 55 Forms of Payoff for Metrics I.e., Which Metrics to Use Low Cost of Collection and Storage Minimal Impact of Collection (morale, disruption, etc.) High Information Communicated One Metric answers Multiple Questions One Metric supports Multiple Goals

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 56 Look for Questions and Metrics that Do Dual Duty Goal 1Goal 2 Q1Q2Q3Q4Q5 M1M2M3M4M5M6M7

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 57 Other Attributes of a Good Set of Metrics Completeness - Addresses All Issues of Concern Value is High relative to Cost of Collection, Storage and Analysis There is an Organizational Framework for Understanding the Importance of Metrics – I.e., people do not sabotage the metrics collection effort – And people do not draw wrong conclusions All Parts of the Organization Benefit Alternative Metrics are Available if Primary Choices are Not Available

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 58 Example: GQM Approach Goal: Improve Efficiency by Reducing the Cost of Rework Question 1: What does rework cost us? Question 2: How many defects are in the software when we release it? Question 3: What are the sources of defects and rework?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 59 Possible Metrics for Question 2 How Many Defects are In the Software when we Release It? Number of Defects in Product -- this is impossible to know -- but we can approximate it with other metrics Number of Known Defects at Release -- incomplete, but perhaps correlated to the total Number of Defects Detected After Release Total Defects over the Life of the Product

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 60 Analyzing Defect Data Release12 Months Retirement Total Defects Known at Release Known after 12 Months Total Defects in Product Life

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 61 Tie Metrics to the Process -- A Measure for Every Phase Phase Metric Require ments Test Plan ning Design Integr ation Coding Staffing X X XXX X Requirements Stability XXX Etc. X Design Complexity X Code Complexity

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 62 Risk: “Running out of memory space” Possible Measures : – Predicted memory space used (update each week or month, depending on risk) – Size of code produced by compiler – Level of experience of average staff Selecting Measures Any risk may result in measures for monitoring Choice depends on anticipated causes of risk

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 63 Another Example Risk Measure Risk: “Not meeting schedule” Cause: Not enough experienced staff Possible Measures: Staffing level Unplanned turnover Average staff experience, etc.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # Recommended Metrics Problem Reports Customer Satisfaction Requirements Stability Rate Charts Earned Value Resource Use Defects Rework

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 65 Problem Reports Complaints Help you Improve Reports should be categorized by such things as: -- Kind of problem -- Severity -- Source of the problem (process step) -- Effort required to correct -- When discovered This can tell you where you need to focus improvement efforts Problem reports are a direct measure of customer satisfaction

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 66 Problem Report Analysis

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 67 Some Issues with Measuring Problem Reports How to handle duplicates How to handle different symptoms caused by the same root problem How to handle reports that are the result of multiple root problems Urgency vs cost vs other factors in deciding on priority

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 68 Customer Satisfaction -- as measured by the customer Surveys Degree of cooperation Do they buy your product? etc. Business survival depends on satisfying the customer better than the competition

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 69 Requirements Stability TBDs + New + Changes Total Requirements

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 70 Requirements Stability Another Way to Display

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 71 Rate Charts These tell you the rate at which something is happening - usually compared with some plan What you measure is something discrete that represents genuine progress toward the objectives of the project If you measure well, rate charts can give you a better handle on where you are than something like “hours spent”

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 72 Rate Charts Example Completion vs. history and plans Today Deadline

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 73 Staffing: actual vs. plan

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 74 Staffing: actual vs. plan

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 75 Staffing: actual vs. plan

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 76 Rate Charts can be used for Many Things Examples: -- Modules designed/coded/tested/integrated -- Requirements designed/coded/tested/integrated -- Problems corrected -- High priority complaints resolved

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 77 Characteristics of Applications for which Rate Charts are a Good Choice Discrete Events Easily Measured Clear Definition of Completion Many Occurrences

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 78 Earned Value Questions: “How much of the work have we actually done?” “Are we likely to meet the deadline?” Metric: “Amount of work actually performed, and projection for duration of project.” Measure: Earned Value = % of total work performed

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 79 Earned Value Uses These Data ACWP (Actual Cost of Work Performed) – How much money has been spent? BCWP (Budgeted Cost of Work Performed) – How much money should have been spent for the work actually performed? BCWS (Budgeted Cost of Work Scheduled) – How much work should have been done by now according to the schedule?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 80 Units “Money” and “work” can be measured in dollars, effort, or other units Effort is usually the easiest way to measure But dollars can be adjusted to include overhead costs and non-labor costs, so it may be more accurate

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 81 Questions You Can Answer with Earned Value Are we getting the work done on schedule? Are we overspending? Will we overrun or under-run the budget? What is a realistic end date? What performance level is required to meet the budget or the deadline?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 82 Earned Value Provides Insight Early in the Project You can tell if there are problems after a few weeks or months You have time to make adjustments You can communicate the problems and their scope using quantitative methods rather than guesswork

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 83 Rate Charts Can Display Earned Value in a Visual Manner Rate charts give a visual picture of how much “value” you have “earned” Consider some examples...

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 84 Requirements Implemented

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 85 A Rate Chart for Iterative Lifecycles

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 86 Earned Value Metrics Quantify Progress and Help Predict and Manage

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 87 Top Level View of Earned Value Metric Earned Value $ Spent vs $ Planned $ Spent vs Thresholds of what should have been spent Budget “Are we overspending?” WBS or Requirements “Are we getting the work done?”

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 88 A Simple Way to do Earned Value This method avoids a lot of the jargon and just focuses on the substance If you understand this method you will find it easier to understand the definitions of earned value metrics found in other places

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 89 Develop a Micro Schedule for the Next Part of the Project Top Level Schedule Micro Schedule for the Next Phase Micro Schedule is the locally- managed schedule, defined by those doing the work

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 90 What Is a Micro Schedule A schedule of small tasks whose duration can be measured in days or a few weeks This represents the work tasks assigned to individuals Have the individuals develop their micro schedule

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 91 Earned Value Micro Schedule Basic Task Estimates TaskEffort PlannedComplete DateResp. (days)(week #) Set Up31Joe Get Specs22Mary Design Output105Pete & Joe Plan Tests36Joe Code57Mary Unit Test38Joe Integrate29Mary Beta Test310Pete Total31

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 92 Convert to Running Total of Effort Planned WeekTotal Effort Planned to be Complete (*) 411 (*) (*) Assumes partial progress on “design output” task

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 93 The Total Planned Effort is the Budgeted Cost of Work Scheduled WeekTotal Effort Planned to be Complete BCWS

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 94 Earned Value Plan Graph (BCWS) Some systems estimate BCWS by drawing a straight line from 0 at start to total project budget at end. BCWS

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 95 Budget Your actual budget should match this plan If the actual budget does not match, you can negotiate what to do before beginning the work – Reduce plan to match available budget – Increase budget to match plan – Somewhere in between – Renegotiate plan with developers

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 96 TaskEffort (days)% CompleteEarned Set Up31003 Get Specs2501 Design Output Plan Tests300 Code500 Unit Test300 Integrate200 Beta Test300 Total316.5 BCWP Earned Value Data for a Given Week

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 97 Earned Value Procedure Each week you compute a new “% complete” value for each task – Perhaps allow only 0 or 100% – Or perhaps allow 50% or 33% or 25%, but you should avoid smaller increments Then compute the total work performed so far Plot the total each week in the plan graph, representing an “actual” line

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 98 Earned Value Typical Graph BCWPBCWS

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 99 Earned Value Applications You can spot deviations from plan and project completion dates You can also plan overtime or other contingency activities You can show your management that you know where you are and are not suffering from the “always 90% done” syndrome

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 100 Earned Value Advantages By using only small tasks, it is easier to tell what is actually complete Developers can use the data to manage their work - it is their plan – The data gives them insight on what they are accomplishing The detailed work sequence does not matter. All completed work is “earned” – Developers can make the right decisions about work sequencing

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 101 ACWP The Third Number Actual Cost of Work Performed is what you have actually spent Whereas BCWP is what you have “earned” or accomplished BCWP = Work Completed (earned value) ACWP = Money Spent (or effort spent)

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 102 Typical Graph Showing Work Behind but On Budget BCWS BCWPACWP

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 103 Typical Graph Showing Work Behind but On Budget This project is probably understaffed.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 104 Typical Graph Showing Work Behind and Over Budget This project is not as productive as planned. ACWP

Slide # 105 Summary of the Three Numbers

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 106 BCWS (Budgeted Cost of Work Scheduled) The amount of effort you had planned to expend (i.e., the amount you budgeted) “How much work should have been done” BCWS

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 107 BCWP (Budgeted Cost of Work Performed) The value produced for the effort you have actually expended “How much work was actually done” BCWP

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 108 ACWP (Actual Cost of Work Performed) The amount of money you have spent so far “What did your work actually cost?” ACWP

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 109 What Can We Tell About the Schedule From These Three Numbers? SV - Schedule Variance SV = BCWP - BCWS SPI - Schedule Performance Index SPI = BCWP / BCWS Negative means behind schedule Less than 1 means behind schedule

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 110 SPI Indicates Whether You are Ahead of or Behind Schedule Ahead of Schedule Behind Schedule

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 111 What Can We Tell About the Cost From These Three Numbers? CV - Cost Variance CV = BCWP - ACWP CPI - Cost Performance Index CPI = BCWP / ACWP Negative means over budget Less than 1 means over budget

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 112 CPI Indicates Whether You are Under or Over Budget Under Budget Over Budget

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 113 Two Useful Numbers BAC - Budget at Completion – The total budget for the project – BCWS at project completion SCHED - Schedule for the project – The total number of weeks or months or whatever

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 114 Estimated Budget At Completion EAC - Estimate at Completion – The estimated actual cost of project EAC = BAC / CPI VAC - Variance at Completion VAC = BAC - EAC Negative means over budget by this much

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 115 Schedule At Completion SAC - Schedule at Completion – The estimated actual schedule of project SAC = SCHED / SPI SVAC - Variance at Completion SVAC = SCHED - SAC Negative means behind schedule by this much

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 116 Thus You Can Tell... How far behind or ahead of schedule you are How far over or under budget you are What your eventual schedule and cost are likely to be You also have an early warning of trouble that you can use to try to fix the underlying problems

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 117 A Handy Graph of SPI and CPI Indicates Your Overall Status SPI CPI

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 118 Reasons for Being Behind Schedule If you are also Under Budget – There is not enough work being done Not enough people People being used for other tasks If you are at or Over Budget – There is a productivity problem Underestimation of the effort Excessive “non-value-added” work

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 119 Resource Use Metrics Staffing on project Staff-hours worked Memory requirements for product CPU capacity required Amount of reused code You can predict beforehand and then measure actuals When actuals deviate from plan, you know that you need to act

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 120 Example of Memory Requirements Metric

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 121 Example of Reuse Measurement

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 122 The Estimation Spreadsheet (Assignments 4/5) Provides Initial Plan Data Staffing levels Headcount Dollars spent Size in Lines of Code Size in Memory Reuse etc.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 123 The Estimation Spreadsheet (Assignments 4/5) Provides Initial Plan Data Periodic updates can be represented by additional rows and/or columns Actuals and history can also be represented by additional rows and/or columns

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 124 Measuring Defects Defects are one of the most valuable things you can measure They indicate fundamental process problems that are causing you to waste time and money and produce lower quality products But people resist measuring defects because they represent “bad news” It takes a cultural change, sometimes, to get defect measurement viewed as a positive thing

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 125 Kinds of Defects to Measure Any product or byproduct can have defects. Requirements analysis defects - collect at inspections and reviews Design defects - collect at design walkthroughs, inspections, reviews Coding defects - collect during peer reviews and tests Testing defects (defects in testing procedures) etc.

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 126 Rework Rework is work you do over because the first time was wrong, unneeded, solved the wrong problem, etc. Every process has some rework, but rework adds cost and time -- and it reflects quality problems Most organizations resist measuring rework because it makes them look bad But rework is an excellent metric for evaluating process health and effectiveness

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 127 Rework Loops Process Step Undetected Defect Several Steps Process Step Defect Detected Rework costs money and time

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 128 Measuring Rework Simple Method: count problems Count problem reports and estimate effort to fix each problem The sum of all the effort is an estimate of total rework Compare with total effort on project This only works if you identify problems, document them as problems, and estimate the work required to fix them More Advanced Method: phase containment (see next slide)

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 129 In-Phase vs. Out-of-Phase Defects Defect Containment Chart Integ rate Test Code Design Rqmts Input Rqmts Design Integr ate TestCode Phase where Defect was Inserted PhaseDetectedPhaseDetected

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 130 Cost of Rework In-phase defects are often not counted as rework -- they can be viewed as a part of the process (more on this in a later course) Out-of-phase defects cost a lot more to fix, in general, and that effort is usually classified as rework

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # Metrics Issues Frequency Synchronization Updating estimates Margin Peak vs. average Units Interplay of resources

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 132 Frequency How Often to Measure Too often results in high cost, disruption of process Too seldom results in failure to see problems soon enough Frequency should depend on degree of risk and cost of measurement

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 133 Synchronization Getting the True Picture

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 134 Update Projections and Estimates

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 135 Margin A Frequently Misunderstood Term What does “leave 50% margin” mean? Code Margin 66 2/ Code Margin

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 136 Peak vs. Average “Software must be able to handle 100 screens per minute” Peak? (Worst Case) Average over Some Interval? How Often to Measure?

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 137 Units Often Subject to Misinterpretation Memory Size: words, bytes, bits??? Program Execution Time: – Hours since shipped? {was it delivered?} – Hours since received by customer? {was it installed?} – Hours since installed? {was it used?} – CPU execution time since installed? CPU Time: cycles, instructions, MIPS, FLOPS, ? Beware of innocent misinterpretations

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 138 Interplay of Resources Few Things Happen in Isolation Sometimes, system resources interact with each other, giving misleading evidence of performance Memory Capacity: 80 cycles per micro- second CPU 50 MIPS I/O Channel 20 MIPS Display 40 MIPS CPU may run at only 20 MIPS because of memory saturation

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 139 Summary 1) Know WHAT you are measuring - attributes, etc. 2) Know WHY you are measuring - goals, risks, etc. 3) Select wisely – Complete - something for each part of the process – Reasonable cost of collection – High information content – Minimal organizational disruption 4) Beware of incorrect interpretations 5) Beware of misuse “Nobody believes statistics anyway.” Adams, The Dilbert Principle

January 20, 2000 CSE SW Project Management / Chapter 10 – Measuring and Monitoring Copyright © , Dennis J. Frailey, All Rights Reserved Slide # 140 End of CHAPTER 10