Project Measurement Source: Practical Software Measurement John McGarry, et.al.

6/19/2007SE _6_19_PSMp1.ppt2 Measurement “If you can’t measure it, you can’t manage it” Tom DeMarco

6/19/2007SE _6_19_PSMp1.ppt3 Fundamentals Don’t try to measure everything Align measures with: Project goals & risks (basic survival mode) Process improvement areas (continual improvement mode) Define measurement program up front Monitor continuously & take action where needed

6/19/2007SE _6_19_PSMp1.ppt4 Applications Improve accuracy of size & cost estimates Improve quality Understand project status Produce more predictable schedules Improve organizational communication Faster, better informed management decisions Improve software processes

6/19/2007SE _6_19_PSMp1.ppt5 Basic In-Process Measurement Examples Schedule Earned Value vs. Planned Value Schedule Variance Development Task completion Actual code completed vs. planned Project End Game Defect Creation vs. Closure Variations: severity System Test % Testing Complete Variations: passed, failed, blocked Test Time / Defect Test Coverage (vs. requirements, white box code coverage)

6/19/2007SE _6_19_PSMp1.ppt6 Process Improvement Measurement Examples Quality Defect density Post Deployment defect density Inspection Effectiveness Defects / inspection hour Estimation Accuracy

6/19/2007SE _6_19_PSMp1.ppt7 Case Study 1: The mature product Project characteristics Release ‘N’ of successful product Mature technology Well understood requirements Tenured team Realistic schedule, adequate staff & budget Project Goals? Project Risks? Development Strategy? Measurement Plan?

6/19/2007SE _6_19_PSMp1.ppt8 Case Study 2: The Schedule Challenged Project Project characteristics Significant amount of new capability “Immovable” end date Mature technology Tenured team Adequate staff & budget Inadequate schedule Project Goals? Project Risks? Development Strategy? Measurement Plan?

6/19/2007SE _6_19_PSMp1.ppt9 Case Study 3: The Technology Challenged Project Project characteristics Several new “bleeding edge” technologies Tenured team Adequate staff & budget Schedule constrained Project Goals? Project Risks? Development Strategy? Measurement Plan?

6/19/2007SE _6_19_PSMp1.ppt10 Why Measure? Support short & long term decision making Mature software organization (CMMI level?) uses measurement to: Plan & evaluate proposed projects Objectively track actual performance against plan Guide process improvement decisions Assess business & technical performance Organizations need the right kind of information, at the right time to make the right decisions

6/19/2007SE _6_19_PSMp1.ppt11 Measurement in Software Lifecycle Plan Do – carry out change Check – observe effects of change Act – decide on additional areas for improvement Repeat Considerations: Cost, schedule, capability, quality

6/19/2007SE _6_19_PSMp1.ppt12 Measurement Psychological Effects Measurement as measures of individual performance Hawthorne Effect Measurement Errors Conscious: rounding, pencil whipping (ie. False data entry) Unintentional: inadvertent, technique (ie. Consistent)

6/19/2007SE _6_19_PSMp1.ppt13 Use of Measures Process Measures – time oriented, includes defect levels, events & cost elements Used to improve software development & maintenance process Product Measures – deliverables & artifacts such as documents includes size, complexity, design features, performance & quality levels Project Measures – project characteristics and execution includes # of developers, cost, schedule, productivity Resource Measures –resource utilization includes training, costs, speed & ergonomic data

6/19/2007SE _6_19_PSMp1.ppt14 Measurement Uses Objective information to help: Communicate effectively Track specific project objectives Identify & correct problems early Make key trade-off decisions Justify decisions

6/19/2007SE _6_19_PSMp1.ppt15 Glossary Entity - object or event (e.g. personnel, materials, tools & methods) Attribute - feature of an entity (e.g. # LOC inspected, # defects found, inspection time) Measurement - # and symbols assigned to attributes to describe them Measure – quantitative assessment of a product/process attribute (e.g. defect density, test pass rate, cyclomatic complexity) Measurement Reliability – consistency of measurements assuming nochange to method/subject Software validity – proof that the software is trouble free & functions correctly (ie. high quality) Predictive validity – accuracy of model estimates Measurement errors – systematic (associated with validity) & random (associated w/ reliability) Software Metrics – approach to measuring some attribute Defect – product anomaly Failure – termination of product’s ability to perform a required function

6/19/2007SE _6_19_PSMp1.ppt16 PSM Measurement Process Measurement Plan Information need – e.g.: What is the quality of the product? Are we on schedule? Are we within budget? How productive is the team? Measurable Concept Measured entities to satisfy need (abstract level: e.g. productivity) Measurement Construct What will be measured? How will data be combined? (e.g. size, effort) Measurement Procedure Defines mechanics for collecting and organizing data Perform Measurement Evaluate Measurement

6/19/2007SE _6_19_PSMp1.ppt17 Measurement Construct Attribute Base Measure Derived Measure Indicator Derived Measure Decision Criteria Measurement method Measurement Function Measurement method Analysis Model

6/19/2007SE _6_19_PSMp1.ppt18 Attributes Attribute Distinguishable property or characteristic of a software entity (Entities: processes, products, projects and resources) Qualitative or Quantitative measure Attribute Base Measure Derived Measure Indicator Derived Measure

6/19/2007SE _6_19_PSMp1.ppt19 Base Measure Measure of an attribute (one to one relationship) Measurement method Attribute quantification with respect to a scale Method type Subjective (e.g. high, medium, low), Objective (e.g. KLOC) Scale Ratio Interval Ordinal Nominal Unit of measurement e.g. hours, pages, KLOC Attribute Base Measure Derived Measure Indicator Derived Measure

6/19/2007SE _6_19_PSMp1.ppt20 Derived Measure Indicator Derived Measure Function of 2 or more base measures Measurement Function Algorithm for deriving data (e.g. productivity = KLOC/developer hours) Indicator Estimate or Evaluation Analysis Model Algorithm / calculation using 2 or more base &/or derived measures + Decision Criteria Numerical thresholds, targets, limits, etc. used to determine need for action or further investigation Attribute Base Measure Derived Measure Indicator Derived Measure

6/19/2007SE _6_19_PSMp1.ppt21 Measurement Construct Examples Productivity Attributes: Hours, KLOC Base Measures:Effort (count total hrs), Size (KLOC counter) Derived Measure:Size / Effort = Productivity Analysis Model:Compute Mean, compute std deviation Indicator:Productivity: mean w/ 2  confidence limits Quality Attributes:Defects, KLOC Base Measures:# Defects (count defects), Size (KLOC counter) Derived Measures:# Defects / Size = Defect Rate Indicator:Defect rate control: baseline mean, control limits & measured defect rate Attribute Base Measure Derived Measure Indicator Derived Measure

6/19/2007SE _6_19_PSMp1.ppt22 More Measurement Construct Examples Coding Base Measure:Schedule (w.r.t. coded units) Derived Measure:Planned units, actual units Analysis Model:Subtract units completed from planned units Indicator:Planned versus actual units complete + variance Attribute Base Measure Derived Measure Indicator Derived Measure

6/19/2007SE _6_19_PSMp1.ppt23 Class Measurement Construct Examples Coding Base Measure: Derived Measure: Analysis Model: Indicator: Attribute Base Measure Derived Measure Indicator Derived Measure

6/19/2007SE _6_19_PSMp1.ppt24 Identify Candidate Information Needs Project Objectives Cost, schedule, quality, capability Risks Prioritize One approach: probability of occurrence x project impact = project exposure e.g. Schedule Budget Reliability Dependencies Product Volatility Measurement Planning

6/19/2007SE _6_19_PSMp1.ppt25 PSM Common Information Categories Schedule & Progress Resources & Cost Product Size & Stability Product Quality Process Performance Technology Effectiveness Customer Satisfaction

6/19/2007SE _6_19_PSMp1.ppt26 PSM Common Information Categories Measurement Concepts Schedule & Progress- milestone dates/completion, EV/PV Resources & Cost- staff level, effort, budget, expenditures Product Size & Stability- KLOC/FP, # requirements, # interfaces Product Quality- defects, defect age, MTBF, complexity Process Performance- productivity, rework effort, yield Technology Effectiveness- requirements coverage Customer Satisfaction- customer feedback, satisfaction ratings, support requests, support time, willingness to repurchase

6/19/2007SE _6_19_PSMp1.ppt27 Select & Specify Measures Considerations Utilize existing data collection mechanisms As invisible as possible Limit categories & choices Use automated methods over manual Beware of accuracy issues (e.g. timecards) Frequency needs to be enough to support ongoing decision making (alternative: gate processes)

6/19/2007SE _6_19_PSMp1.ppt28 Measurement Construct Information Need Measurable Concept Relevant Entities Attributes Base Measures Measurement Method Type of Method Scale Type of Scale Unit of Measurement Derived Measures Measurement Function Indicator Analysis Model Decision Criteria

6/19/2007SE _6_19_PSMp1.ppt29 Project Measurement Plan Template (from PSM figure 3-10, p 56) Introduction Project Description Measurement Roles, Responsibilities & Communications Description of Project Information Needs Measurement Specifications (i.e. constructs) Project Aggregation Structures Reporting Mechanisms & Periodicity

6/19/2007SE _6_19_PSMp1.ppt30 Cycle 1 Measurement Plan Identify cycle 1 risks & information needs Pick 4 measures & create measurement constructs Document in a measurement plan per previous slide