Download presentation
Presentation is loading. Please wait.
1
12 Steps to Useful Software Metrics
Westfall The Team Linda Westfall
2
What are Software Metrics?
Applied To Measurement Based Techniques Software Processes, Products & Services To Supply Engineering & Management Information To Improve
3
Entity: Attribute: Mapping:
Measurement Defined Entity: Attribute: Features & Properties Mapping: Numbers & Symbols
4
Two Historic Schools of Thought
Collect data on everything -- then find meaning Implement a random selection of metrics Jeopardy approach to metrics -- start with the answer and try to guess the question
5
Step 1 - Identify Metrics Customers
Who needs the information? Who’s going to use the metrics? ? If the metric does not have a customer -- do not use it.
6
Task goals (entry & exit criteria)
Step 2 - Target Goals Organizational goals Be the low cost provider Meet projected revenue targets Project goals Deliver the product by June 1st Finish the project within budget Task goals (entry & exit criteria) Effectively inspect software module ABC Obtain 100% statement coverage during testing
7
Frustrations Goals Project manager: our estimated project schedules always end up being way off. Functional manager: if we have another late delivery... Engineer: too much overtime -- not enough time to do things right Test manager: no time left to test product before it is scheduled to ship Improve the schedule estimation process
8
Step 3 - Ask Questions Goal: Maintain a high level of customer satisfaction What is our current level of customer satisfaction? What attributes of our products and services are most important to our customers? How do we compare with our competition?
9
Turning Questions into Lower Level Goals
High level goal: Ship only defect free software Questions: Is software adequately tested? How many defects are still undetected? Are all known defects corrected? Lower level goal: Ensure all known defects are corrected before shipment
10
Select metrics that provide information to help answer the questions
Step 4 - Select Metrics Select metrics that provide information to help answer the questions Be practical, realistic, pragmatic Consider current engineering environment Start with the possible Metrics don’t solve problems -- people solve problems Metrics provide information so people can make better decisions
11
Selecting Metrics Goal: Ensure all known defects are corrected before shipment
12
Role of Measurement Understand Control Evaluate Predict Processes,
Products & Services Evaluate Predict
13
Metrics Objective Statement Template
To understand evaluate control predict the attribute of the entity in order to goal(s) Example - Metric: % defects corrected To the in order to % defects found & corrected during testing ensure all known defects are corrected before shipment evaluate
14
Benefits Clearly defined metric objectives: Ensures a well defined metric based on customer goals Eliminates misunderstandings Communicates need Provides requirements statement
15
Step 5 - Standardize Definitions
Open User Open Developer
16
Standardize Definitions
Select definitions from the literature that match your organizational goals Use them as a basis for creating your own definitions Apply them consistently Include them in an appendix to each metrics report
17
Models for code inspection metrics Metric Primitive Models:
Step 6 - Choose a Model Models for code inspection metrics Metric Primitive Models: Lines of Code Inspected = loc Hours Spent Preparing = prep_hrs Hours Spent Inspecting = in_hrs Discovered Defects = defects Other Metric Models: Preparation Rate = loc / prep_hrs Inspection Rate = loc / in_hrs Defect Detection Rate = defects / (prep_hrs + in_hrs)
18
If you are miles away from your target
Keep It Simple If you are miles away from your target -- it doesn’t make sense to measure in millimeters. Questions: Does it give us more information than we have now? Is that information of practical benefit? Does it tell us what we want to know?
19
Step 7 - Establish Counting Criteria
Lines of Code Variations in counting No industry accepted standard SEI guideline - check sheets for criteria Advice: use a tool
20
Counting Criteria - Effort
What is a Software Project? When does it start / stop? What activities does it include? Who works on it?
21
Step 8 - Decide On Decision Criteria
Establish Baselines Current value Problem report backlog Defect prone modules Statistical analysis (mean & distribution) Defect density Fix response time Cycle time Variance from budget (e.g., cost, schedule)
22
Deciding “what’s good”
Establishing Goals Deciding “what’s good” Customer requirements Metrics literature Benchmarking Own best results Avoid the roller coaster effect Decrease variance Baseline Move the mean
23
Step 9 - Define Reporting Mechanisms
24
Metrics Reporting Timing Delivery Mechanism Distribution Availability
Database Data Extraction Snapshot Reporting Delivery Mechanism Distribution Availability
25
Step 10 - Determine Additional Qualifiers
A good metric is a generic metric Additional qualifiers: Provide demographic information Allow detailed analysis at multiple levels Define additional data requirements
26
Additional Qualifier Example
Metric: software defect arrival rate Release / product / product line Module / program / subsystem Reporting customer / customer group Root cause Phase found / phase introduced Severity
27
? Step 11 – Collect Data What data to collect?
Metric primitives Additional qualifiers Who should collect the data? The data owner Direct access to source of data Responsible for generating data Owners more likely to detect anomalies Eliminates double data entry ?
28
Examples of Data Ownership
29
Step 12 – Consider Human Factors
The People Side of the Metrics Equation How measures affect people How people affect measures “Don’t underestimate the intelligence of your engineers. For any one metric you can come up with, they will find at least two ways to beat it.” [unknown]
30
Use metrics as a “stick”
Don’t Use metrics as a “stick” Measure individuals Ignore the data Use only one metric Cost Quality Schedule
31
Select metrics based on goals Focus on processes, products & services
Question 1 Question 2 Question 3 Question 4 Metrics 1 Metric 2 Metric 3 Metric 4 Metric 5 [Basili-88] Provide feedback Feedback Data Data Providers Metrics Focus on processes, products & services Processes, Products & Services Obtain “buy-in”
32
12 Steps to Useful Software Metrics
Step 1 - Identify Metrics Customers Step 2 - Target Goals Step 3 - Ask Questions Step 4 - Select Metrics Step 5 - Standardize Definitions Step 6 - Choose a Model Step 7 - Establish Counting Criteria Step 8 - Decide On Decision Criteria Step 9 - Define Reporting Mechanisms Step 10 - Determine Additional Qualifiers Step 11 - Collect Data Step 12 - Consider Human Factors
Similar presentations
