1. SAM Executive Seminar
Software Measurement

2. What is “Performance Measurement”?
Performance Measurement: is the assessment of the efficiency and effectiveness of IT in support of an organization’s missions, goals, and quantitative objectives through the application of outcome-based, measurable, and quantifiable criteria, compared against an established baseline to activities, operations, and processes US DoD Guide for Managing IT as an Investment
Rev 1.1

3. Program/Project Level
Levels of Measurement
External Oversight
Results
Enterprise Level
Missions, Goals
and Objectives
Results
Functional Level
Functional
Requirements
and
Performance
Measurement
Framework
Results
Program/Project Level
Linkage of Measures
Rev 1.1

4. Spectrum of Performance Measurement
Less
Detail
More
Enterprise Level: focus is on on overall mission results. Measurement information is used on a cyclical (e.g., annual or quarterly) basis to choose policy directions and make mission decisions. Heavily involved in investment allocation of a project.
Functional Level: focus is on unit results where information is needed to manage and improve operations. Measurement information is used on a periodic (e.g., quarterly or monthly) basis to report on performance of major organizational functions that cross multiple programs/projects or acquisitions. Heavily involved in selection, evaluation, and requirements approval of a project.
Program/Project Level: focus is on activity and task information needed to make and execute tactical project and program management resource allocation decisions. Measures are performance oriented. In addition to use by the project manager and staff, these measures are combined, synthesized, and reported to the functional level. Heavily involved in day-to-day project management activities.
Rev 1.2

5. Efficiency vs. Effectiveness
Efficiency: this criteria demonstrates that an organization is employing the best use of available resources. For example:
Are efforts completed within estimates?
Were resources expended optimally?
Effectiveness: this criteria demonstrates that an organization is doing the right things. For example:
Achieving missions and goals
Generating satisfied customers
Producing work of high quality
Rev 1.1

6. What Program Managers Want to Know
metrics!
Is there really a problem?
What is the scope of the problem?
What is causing the problem?
Are there related problems?
Can I trust the data?
What should I expect--What will happen?
What are my alternatives?
What are recommended courses of action?
When can I expect to see results?
1. This is a revised/edited list of items taken from a briefing on the Practical Software Measurement (PSM) given at the DSMC in June 1996 to the JLC’s Joint Group on Systems Engineering . The briefing was a recap of key PSM points and consisted for the most part of slides taken from figures in the PSM manual, vers 2.2.
2. The same briefing noted that version 3.0 of the PSM would be completed by 9/97 and should include materials on software estimation and structured issues analyses. A PSM “Vision and Strategy” is to (1) ID a transition sponsor, (2) establish service/agency PSM focal points, (3) establish a PSM steering committee, (4) expand DoD and industry participation, (5) extend “IPT” approach and (6) continuously improve PSM products.
3. Inasmuch as other services have metrics policies, the “supplemental information” portion of the PSM briefing contained a comparison chart that showed the value added by the PSM approach to DoD, SBP, OSD, STEP, USAF and SEI metrics proposals. The comparison showed the PSM either augments policy or guidance or fulfills a requirement established by policy or guidance within each of these metrics fiefdoms. It is unclear at this point, however, which, if any of the existing OSD/service metrics policies the PSM will replace. It is an excellent, clear summary that provides valuable insight irrespective of service politics. Another, non-government publication is the SPC’s Software Measurement Guidebook (SPC CMC) and the IDA Study “Survey of Metrics in DoD and Industry” (IDA P-2996, April 1994)
2.3

7. Categories of Project-Level Metrics
“How do you know?”
“What does that mean?”
“Can you show me?” PM
“Common-Sense” Metrics
Management
Metrics
Progress
Size & Cost Status
Resource Margins
Volatility, etc.
Metrics
Metrics
Metrics
Metrics
Quality
Metrics
Error Density
Complexity
Reliability, etc.
Metrics
Metrics
Metrics
Process
Metrics
CMM (SEI)
SDCE (USAF)
SPICE (ISO), etc.
Even more Metrics!
Rev 2.2

8. Some Key Software Measurement Principles*
Software measures should be driven by program-specific issues and objectives
The developer’s software process defines how the software is actually measured
Collect and analyze low level data
Use the measurement process as a basis for objective communications
Use a structured analysis process to trace measures to decisions
Measurement results must be considered in the context of other information from the program
The PM must have a measurement analysis that is independent of the software developer’s
Software measurement must be an integral part of program management throughout the life cycle
The primary measurement analysis focus should be at the single program level
PM Note: A myth exists that program managers can just collect data and plot graphs to form an effective measurement process. Although possibly true in a static, precedented software development environment, such not the case for the majority of DoD software acquisitions. The measurement must be dynamic because of the constantly changing issues and complexities in DoD contracting
*Courtesy of the PSM’s Practical Software Measurement Guidebook
Rev 2.1

9. Examples of Software Measures
Software Size
Software Staffing
Software Complexity
Software Progress
Problem Report/Change Report Status
Build Release Content
Computer Hardware Resource Utilization
Milestone Performance
Scrap/Rework
Effect of Software Reuse
Requirements Volatility
Note: This is a set of management indicators shown here that might be used on a software development project....There is no intent to impose these indicators or to preclude using any other ones...the acquirer picks a subset driven by program risks and issues and in accordance with service policies.
Rev 4.4

10. What’s a “METER”?

11. Example of Mapping Project Information Needs
Practical Software and Systems Measurement
PSM Version 5.0c, 11
PSM  All rights reserved.
Example of Mapping Project Information Needs
Main Concept:
Project-specific Information Needs are “local” variations of the common Information Categories.
Key Points:
Each project needs to identify their own project-specific Information Needs, and they should use their own terminology when describing these needs.
Information Needs may be known at the beginning of the project, or they may surface at any time during the project. This example shows both types of Information Needs.
The red items under "Project Specific Info. Needs" are high-priority issues for which information is needed.
Project-specific Information Needs can usually be grouped together and mapped to the seven PSM Information Categories.
In this case, each project-specific Information Needs has been mapped to one of four PSM common Information Categories. In actual practice, a single information need might relate to multiple Information Categories.
NOTE: The mapping isn’t necessarily “many-to-one” but may be “many-to-many”. For example, Concurrent Activities relates to Resources and Cost as well as to Schedule and Progress.
Transition:
Let’s work on an exercise to gain some experience in identifying project-specific Information Needs and mapping them to the PSM common Information Categories...

12. Categories, Concepts, and
Practical Software and Systems Measurement
PSM Version 5.0c, 12
PSM  All rights reserved.
PSM
Mapping of
Information
Categories, Concepts, and
Measures
Main Concept:
This slide and the next show the mapping of PSM Information Categories to Measurable Concepts, and then to Prospective Measures. You have a large version of this table in the "Large Tables" section of your Student Notebook.
Key Points:
NOTE: Tell students to refer to Part 3 of the PSM Guidebook for details on the Info Needs, Measurable Concepts, and Measures. (Version 4 of the Guidebook uses the terms Issues, Categories, and Measures - this is in Part 2.)
The mapping from Information Category to Measurable Concept to Measure is not necessarily two-dimensional, as the table suggests—Measures may apply to more than one Information Category or Measurable Concept. For example, Lines of Code is a measure related to the concept of Physical Size and Stability, but may also be a component of the Productivity measure, which relates to the concept of Process Efficiency.
PSM’s 60+ prospective measures are not all inclusive. These measures have been identified by the PSM Working Group as having been practiced and have been found to be valuable. Measurement users typically add to this collection over time.
Note that the table is a “catalog“—you don’t have to “buy” everything available. Most projects will not implement all of these measures, and may only implement a few, based on their Information Needs and process capabilities.
Transition:
The next slide is a continuation of the table…

13. Categories, Concepts, and
Practical Software and Systems Measurement
PSM Version 5.0c, 13
PSM  All rights reserved.
PSM
Mapping of
Information
Categories, Concepts, and
Measures
PSM
Mapping of
Information
Categories, Concepts, and
Measures
Main Concept:
This slide and the next show the mapping of PSM Information Categories to Measurable Concepts, and then to Prospective Measures. You have a large version of this table in the "Large Tables" section of your Student Notebook.
Key Points:
NOTE: Tell students to refer to Part 3 of the PSM Guidebook for details on the Info Needs, Measurable Concepts, and Measures. (Version 4 of the Guidebook uses the terms Issues, Categories, and Measures - this is in Part 2.)
The mapping from Information Category to Measurable Concept to Measure is not necessarily two-dimensional, as the table suggests—Measures may apply to more than one Information Category or Measurable Concept. For example, Lines of Code is a measure related to the concept of Physical Size and Stability, but may also be a component of the Productivity measure, which relates to the concept of Process Efficiency.
PSM’s 60+ prospective measures are not all inclusive. These measures have been identified by the PSM Working Group as having been practiced and have been found to be valuable. Measurement users typically add to this collection over time.
Note that the table is a “catalog“—you don’t have to “buy” everything available. Most projects will not implement all of these measures, and may only implement a few, based on their Information Needs and process capabilities.
Transition:
The next slide is a continuation of the table…

14. Categories, Concepts, and
Practical Software and Systems Measurement
PSM Version 5.0c, 14
PSM  All rights reserved.
PSM
Mapping of
Information
Categories, Concepts, and
Measures
PSM
Mapping of
Information
Categories, Concepts, and
Measures
(continued)
Main Concept and Key Points:
[continued from last slide]
Transition:
The next slide illustrates the mapping of project-specific Information Needs to the common Information Categories…