1. Verifying – Evaluating Software Estimates
John A. Long
PRICE Systems

2. Most Complex systems in procurement by DOD are Software Intensive
Problem Statement
Most Complex systems in procurement by DOD are Software Intensive
Software is the highest risk component in DOD programs
1995 Standish Survey polled over 800 software developers
52.7% of all projects were completed but incurred cost and schedule overruns
Challenged projects were delivered with only 61% of originally specified functionality
Source: 1995 Standish Survey

3. Overview
Types of Estimates
Evaluation Process
Metrics
Measures
Productivity
Summary

4. Types of Software Estimates
Budgetary
Proposal
Bid to Win?
Deliver within proposed schedule?
Cost to Complete
Cost Growth
Schedule growth
Each type of estimate is reviewed using different techniques and scrutiny

5. Evaluation Process
Bid to Win
Effort only – Does the bottom line cost meet expected number
Only One Parameter Evaluated using this Process
Three Key Parameters Must be Evaluated
Price
Schedule
Quality

6. Number of Lines of Code Produced Per Hour
Evaluation Process
Metrics
Provide Insight Into
Cost/Schedule (Performance)
Process
Quality
Metric Example
Number of Lines of Code Produced Per Hour
Number of Defects per Thousand Lines of Code
Metrics are derived using Software Measures

7. Evaluation Process
Software Measures
Actual Data Collected and Analyzed to form a Metric
PRICE S Input Parameter – APPL is a Measure
Metrics can be Integrated with Software Parametric Cost Model
Should be based on Past History of Completed Programs

8. Metrics
Key is to Derive Metrics from Similar Software Measures
Software Size
As size increases, productivity decreases
Thru-Put Speed
Timing issues/problems reduce productivity
Functional Difficulty of the Code
Productivity decreases as software functional difficulty increases

9. Example Productivity Metrics
Source: Software Productivity Since 1970, by David Longstreet

10. Metrics – Can Aid Evaluation of
Staff Size
Software Department Cost
Software Development Schedule
Program Performance versus Program Plans
Process Improvement
Risk of Achieving Program Success

11. Metrics Cannot Estimate
Software Size
SLOC
Function Points
Objects

12. Objective Software Measures Collected Performance Product Size
Metric Development
Objective Software Measures Collected
Performance
Schedule and Cost
Product Size
Product Quality
Development Process
Process Improvement
Technology
Customer Satisfaction

13. Collect Objective Software Measures SLOC Developed Total Hours
Metric Development
Collect Objective Software Measures
SLOC Developed
Total Hours
Staff Size
Number of Defects
Requirements
Design
Code
Test

14. Metric Development
Normalize Data into Meaningful Metrics
SLOC Per Hour
Defects Per Thousand SLOC
Analyze Results
Why was Productivity Higher for Program XYZ
Expect Differences

15. Reasons for Higher Productivity Measures
Functional Complexity of Code
APPL
Process (Code Generations)
Modification versus New Development
Different Functional Cost and Development Phases included in Software Measure
SLOC versus Function Points

16. Sample Performance Metric
Source: Software Engineering Baselines, Kaman sciences Corporation, July 1996

17. Metric Implementation
Management Buy In
Implemented at the Correct Level to Evaluate Proposal or Product Status
Avoid Metrics that do not Add Value to evaluation process
Understandable Definitions of each Metric
Must be used to Evaluate Product Status, Not Individuals

18. Summary
Metrics can be used to evaluate soundness of software estimate
Metrics should be derived from past history
Metrics Work very well with Parametric Cost Models