1. Software Metrics/Quality Metrics
Software “Quality” Metrics:
Product
Pre-Release and Post-Release Process
Project
Data Collection
Product
Characteristics
Project
Characteristics
Process
Characteristics

2. Software Metrics Software Product Pre-Release & Post Release Processes
All the deliverables
Focus has been on Code, but interested in all artifacts
Product metrics include concerns of complexity, performance, size, “quality,” etc.
Pre-Release & Post Release Processes
Focus has been on Pre-Release processes
Easiest staring point for metrics – Testing and the number of “bug” found
Process metrics used to improve on development and support activities
Process metrics include defect removal effectiveness, problem fix response time, etc.
Project
Cost
Schedule
HR staffing and other Resources
Customer Satisfaction
Project metrics used to improve productivity, cost, etc.
Project metrics include cost such as effort/size, speed such as size/time, etc.
Project and Process metrics are often intertwined.
Will talk about this
more
Function point

3. Product Quality Metrics
What are all the deliverables ?
Code and Help Text
Documentation (function, install, usage, etc. in requirements & design specifications)
Education (set-up/configure, end-user, etc.)
Test Scenarios and Test Cases
Quality Question : (mostly intrinsic to the product but affects external customer satisfactions )
When/where does it fail; how often
how many; defect rate

4. GQM (one more time from Basili)
A reminder on generating measurements:
In coming up with metrics think of GQM
What’s the goal
What’s the Question
What’s the metric
Goal: is Improved Quality
Question: What is the Post Release Defect Rate?
Metric: Number of problems found per user months

5. Some Definitions of Error to Failure
Error – human mistake that results in incorrect software (one or more fault or defect)
Defect or Fault – a mistake in the software product that may or may not be encountered
Problem – a non-functioning behavior of the software as a result of a defect/fault in the product.
Note that an error can cause one or more defects, and a defect can cause one or more problems. But a problem may never surface even if there is a defect which was caused by a human error.

6. When/Where Do Product Failures Occur
When/Where are somewhat intertwined
Right away – e.g. happens at install
Sometimes - e.g. happens at initialization-configuration
Sometimes – e.g. happens at certain file access
Generalized Metric
Mean time to failure (MTTF)
Difficult to assess
What should be the goal (8 hours, 30 days, 6 months), or should we just say --- “lessen the failure rate”?
Hard to test for and analyze (especially- prod. education, doc., etc.)
Applies better for simple logic (like stays up for z amount of time)
Meantime to failure for install problem should probably be close to 0

7. Product Defects and Defect Rate
Most of the metric has been asked in terms of code but should be more inclusive:
Defect Volume: How many defects (for the complete product - not just for code)
Defect Rate = defects/(opportunity of defect)
Defects of all kind or by type (e.g. code, test cases, design, etc.)
Defects by severity (not quite a rate – more by category)
Opportunity of defect by (* also used to assess volume) :
Code : loc, function point, module
Documentation : pages, diagrams
Education or training: # of power point slides (doc) or amt. of time (delivery)

8. Code Defect Opportunity (LOC)
Using Lines of code (loc) “problems”
Executable, non-executable (comments)
Test cases and scaffolding code
Data and file declaration
Physical line or logical line
Language difference (C, C++, assembler, Visual Basic, etc.)

9. Possible Code Defect Rate Metrics
Often used :
Valid Unique Defect per line of executable and/or data code released(shipped)
IBM’s total valid unique defects / KSSI
Total valid unique defects / KCSI (only changed code)
Valid Unique Defect of “high severity” per line of executable and/or data code released (shipped)
What about all “in-line comments”; should they not count ? These provide opportunity of defects too. (especially for pre and post condition specifications)
What about Help text ?

10. Product Quality Metric (User View)
Defect rate is not as useful from user perspective:
What type of problems do users face?:
screen interface
data reliability/(validity)
functional completeness
end user education
product stability - crashes
error message and recovery
Inconsistencies in the handling of similar fucntionalities
How often are these types of defect encountered?
---- counted with -- MTTF -- means more to users?
Possible metric is : Problems per User Month(PUM)
user month is dependent on length of period and the number of users (this takes some tracking effort)
More Broader Customer Satisfaction issues
CUPRIMDSO – capability, usability, performance, rel. etc. (IBM),
FURPS – functionality, usability, reliability, etc. (HP)

11. Begin Function Point
Separate Segment

12. Function Point (product size or complexity) metric
Often used to assess the software complexity and/or size
May be used as the “opportunity for defect” part of defect rate
Started by Albrecht of IBM in late 70’s
Gained momentum in the 90’s with IFPUG as software service industry looked for a metric
Function Point does provide some advantages over loc
language independence
don’t need the actual lines of code to do the counting
takes into account of many aspects of the software product
Some disadvantages include :
a little complex to come up with the final number
consistency (data reliability) sometimes varies by people

13. Function Point Metric via GQM
Goal : Measure the Size(volume) of Software
Question: What is the size of a software in terms of its:
Data files
Transactions
Metrics:
amount/difficulty of “Functionalities” to represent size/volume
consider Function Points ---- (defined in this lecture)
What kind of validity problem might you encounter?
– “construct”: applicability, “predictive”: relational ; “content” : coverage?

14. FP Utility Where is FP used?
Comparing software in a “normalized fashion” independent of op. system, languages, etc.
Benchmarking and Prediction:
size .vs. cost
size vs development schedule
size vs defect rate
Outsourcing Negotiation

15. Methodology Identify and Classifying: Evaluation of Complexity Level
Data (or files/tables)
Transactions
Evaluation of Complexity Level
Compute the Initial Functional Point
Re-assess the range of other factors that may influence the computed Functional Point and Compute the Function Point

16. 1) Identifying & Classifying 5 “Basic Entities”
Data/File:
Internally generated and stored (logical files and tables)
Data maintained externally and requires an external interface to access (external interfaces )
Transactions:
Information or data entry into a system for transaction processing (inputs)
Information or data “leaving” the system such as reports or feeds to another application (outputs)
Information or data displayed on the screen in response to query (query)
Note: - What about “tough” algorithms and other function oriented stuff?
(We take of that separately in a separate 14 “Degree of Influences”)

17. 2) Evaluating Complexity
Using a complexity table, each of the 5 basic entity is evaluated as :
low
average
high
Complexity table uses 3 attributes for decisions
# of Record Element Types (RET): e.g. employee data type, student record type ---- # of file types
# of unique attributes (fields) or Data Element Types (DET) for each record : e.g. name, address, employee number, and hiring date would make 4 DETs for the employee file
# of File Type Referenced (FTR): e.g an external payroll record file that needs to be accessed

18. 5 Basic Entity Types uses the RET, DET, and FTR for Complexity Evaluation
For Logical Files and External Interfaces (DATA):
# of RET DET DET DET
Low Low Ave
Low Avg High
Avg High High
For Input/Output/Query (TRANSACTIONS):
# of FTR DET DET DET
Low Low Ave
Low Avg High
Avg High High

19. Example
Consider a requirement: ability or functionality to add a new employee to the “system.”
(Data): Employee information involves, say, 3 external file that each has a different Record Element Types (RET)
Employee Basic Information file has employee data records
Each employee record has 55 fields (1 RET and 55 DET) - AVERAGE
Employee Benefits records file
Each benefit record has 10 fields (1 RET and 10 DET) LOW
Employee Tax records file
Each tax record has 5 fields ( 1 RET and 5 DET) LOW
(Transaction): Adding a new employee involves 1 input transaction which involves 3 file types referenced (FTR) and a total of 70 fields (DET). So for the 1 input transaction the complexity is HIGH

20. Function Point (FP) Computation
Composed of 5 “Basic Entities”
input items
output items
inquiry
master and logical files
external interfaces
And a “complexity level index” matrix :
Low
Average
High
Input 3 4 6
Output 4 5 7
Inquiry 3 4 6
Logical files 7 10 15
Ext. Interface 5 7 10

21. 3) Compute Initial Function Point
Σ [Basic Entity x Complexity Level Index]
all basic entities
Continuing the Example of adding new employee:
- 1 external interface (average) = 7
- 1 external interface (low) = 5
- 1 input (high) = 6
Initial Function Point = 1x7 + 1x5 + 1x5 + 1x6 = 23

22. 4) More to Consider
There are 14 more “Degree of Influences” (DI) on a scale of :
data communications
distributed data processing
performance criteria
heavy hardware utilization
high transaction rate
online data entry
end user efficiency
on-line update
complex computation
reusability
ease of installation
ease of operation
portability (supports multiple sites)
maintainability (easy to change)

23. Function Point Computation (cont.)
Define Technical Complexity Factor (TCF):
TCF = (.01 x DI )
where DI = SUM ( influence factor value)
So note that ≤ TCF ≤ 1.35
Function Point (FP) = Initial FP x TCF
Finishing the Example:
Suppose after considering 14 DI’s, our TCF = 1.15, then:
Function Point = Initial FP x TCF = 23 x 1.15 = 26.45

24. Defect Rate: Defects/FP by CMM Levels
C. Jones estimated defect rates by SEI’s CMM levels through the maintenance life of a software product:
CMM Level 1 organizations – 0.75 defect/FP
CMM Level
CMM Level – 0.27
CMM Level – 0.14
CMM Level – 0.05
Be careful of this type of claims – use it with caution

25. End Function Point
Separate Segment

26. Pre-Release Process Quality Metrics
Most common one is from testing (Defect Discovery Rate)
defects found (by severity) per time period ( per dev. phase)
Compare “defect arrivals” by time by test phase
looking for “stabilization” (**what would the curve look like?*)
looking for a decreasing pattern
Compare number of defects by products
those with high number of problems found during pre-release tend to be “buggy” after release (interesting phenomenon)
Other Pre-Release quality metric: Defect Removal Effectiveness (e.g. Via inspection)
defects removed / ( total latent defects)
latent defects are estimated : how estimated? --- go back later with defects found in the field

27. Post-Release Product and Process
# of Problems per Usage-Month (# of PUM)
Post-Release “Fix Process”:
Fix Quality = Number of Fix bugs/ Total number of fixes
Very sensitive if fix quality is not close to zero
Post-Release Process Quality
Problem backlog = total # of problems unresolved
by severity
by arrival date
Problem Backlog Index = # of problems resolved / # of arrivals per some time period such as week or month
Average Fix Response Time ( from problem open to close )
These metrics are usually compared with a goal:
average response time on severity 1 problem is 24 hours
problem backlog index is between 1.3 and .8 (.8 may be problem!)

28. Decide on what Metrics are to be used
Collecting Data
Decide on what Metrics are to be used
measuring what (validity of measure)
what’s the goal (validity of measure)
Decide on how to collect the data
clearly defining the data to be collected
assure the recording is accurate (reliability)
assure the classification is accurate (reliability/validity)
Decide on tools to help in the collection
source code count
problem tracking

29. Data Collection Methodology (Basili & Weiss)
Establish the goal of the data collection
Develop a list of questions of interest
Establish data categories
Design and test data collection mechanism (e.g. forms)
Collect and check the reliability data
Analyze the data