1. Information Systems Acquisition Measurement & Analysis
Defense Acquisition University
Intermediate Information Systems Acquisition
IRM 202
14 December 2014

2. IRM 202 Measurement & Analysis Notes
Slides are provided for each 19.1 and 19.2 M&A ELO allocated to IRM 202
The initial slides addresses the measurement & analysis “overview” material intended to be integrated into IRM 202 Lesson 4 – Planning
A number of amplifying “practicum slides” are embedded in the presentation - they are marked as such – they go into the student materials for the practicum
These are added material that may be helpful while students do the measurement piece of the practicum
Added instructor notes and more detailed context specific material may be generated to support IRM 202 practicum revision

3. Slides for Practicum 1

4. IT Measurement & Analysis Provides:
Objective insight into project, portfolio, and enterprise performance
Fact-based information to identify and manage risk
Early detection and resolution of emerging problems
Objective support for management and technical tradeoff decisions
Objective team, project, and enterprise communications
The ability to defend and justify decisions
Data to satisfy mandated DoD reporting requirements
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let’s look at an example….
Part 1

5. Measurement Main Concept:
An example of the importance of data versus information.
Key Points:
This data is the data that was used when the decision was made to launch the space shuttle Challenger.
This chart represents the number of O rings that have distress over a range of air temperature. It omits all flights that did not have any problems. It is not very helpful, and did not identify any issue with launching at the temperature of 31 degrees (the temperature on launch day).
Low level personnel working on O rings stated that there should be no launch below 55 degrees. Information went through 5 management levels before decision was made.
Transition:
Now let’s see what happens when the data is charted in another way…
Part 1

6. Decision Information temperature at launch Main Concept:
The right information at the right time is what PSM is aiming to achieve. Information is the key. Not just data.
Key Points:
On this chart, all flights have been added, along with a trend line to represent the data at lower temperatures.
No launches have been attempted below 50
No launches have zero (0) O-ring failures below 65
The forecast for the last flight of the Challenger was 31 degrees.
Based on this data should they have flown?
Could problems have been expected?
Transition:
In this course, we will discuss how we can generate information that can be used for decision making. Let’s look at the fundamental principles…
Part 1

7. Fundamental Measurement Principles
What you measure is driven by what you need to know - information needs
The existing technical and management infrastructures define what you are able to measure - process integration
The environment you work in drives how the measurement results are interpreted - decision context
Action must be taken to realize any benefit from measurement - fact based decision making
Main Concept:
PSM defines a flexible measurement process that has a goal of providing insight into a project. PSM was designed around the 4 key principles shown here.
Key Points:
Information Needs - Make sure what you measure is based on your information needs. Your information needs can change as you go through different project phases or your enterprise matures.
Process Integration - You have to work within your existing infrastructure. Use your measurement system to determine what improvements to make later.
Decision Context - The management environment in which you work provides the context which will interpret and use the information provided by the measurement results.
Decision making - You must make decisions and take the appropriate actions based on the results to realize any benefit. Without action measurement doesn’t provide any value.
Transition:
What exactly are information needs? They are….
Part 1

8. Information Needs
What an IT decision maker needs to know is driven by program and enterprise objectives, issues, and acquisition/technical mandates - these are called information needs
Objectives are project goals or requirements
Issues are areas of concern that could impact the achievement of an objective and include risks, problems, and lack of information
Acquisition/technical mandates come from policy (e.g series and the Defense Acquisition Guidebook) and from mandates (e.g. OMB Circular A-11 Part 7, Section 30)
The current set of project/enterprise prioritized information needs define what measures are implemented and the focus of the analysis
With limited program resources, measurement efforts must be focused on the critical information needs - these change over the course of the program
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

9. Selecting and Specifying Measures
A key aspect of the measurement process is defining and aligning the measures that support the decision maker’s information needs
The measures that are actually implemented are a synthesis of user information needs and the ability to generate the measurement data.
The implemented measurement set is directly influenced by the instantiated program technical and management processes
The decision maker needs to establish an “audit trail” between what is measured and the decisions that are made
This is the basis for “fact-based”, justifiable decisions
Every decision maker in the program/enterprise structure has a unique set of information needs
The same measurement data can be analyzed in different ways to address different information needs
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
The main categories of information categories that most programs experience are….
Part 1

10. Information Categories
Program IT information needs are categorized into general information categories - Most programs implement measures in these categories
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
How do you transition from your program information needs to the measures?....
Part 1

11. Measurement Analysis
Measurement analysis focuses on taking the measured IT parameter results and transforming them into information products that relate directly to one or more information needs
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at the measurement construct….
Part 1

12. Measurement Construct - Levels
Information
Product
Indicator
Derived
Measures
Measurement
Construct
Main Concept:
In the three-level illustration of the information model, we saw that the Measurement Construct was the middle layer. The Measurement Construct itself is a multi-level structure.
Key Points:
The three levels of the Measurement Construct each represent a different level of information measure detail:
Base Measures, or the raw data collected, are at the bottom level of the Measurement Construct.
Derived Measures, which are combinations of Base and/or Derived Measures, are at the middle level.
Indicators, which provide a means for answering a question about a Measurable Concept, are at the top level.
Transition:
Now we will look at the information model in more detail…
Base
Measure
Attribute
Part 2

13. Measurement Information Model
Product
Information Needs
Estimate or evaluation that provides a basis for decision making
Interpretation
Indicator
Analysis
Model
Algorithm combining measures and decision criteria
Measurement Information Model
Derived
Measure
Derived
Measure
Quantity defined as a function of two or more measures
Measurement
Function
Algorithm combining two or more
base measures
Base
Measure
Base
Measure
A measure of a single attribute
by a specific method
Main Concept:
Base Measures, Derived Measures, and Indicators each have two key components, a value and a procedure for assigning that value. (Actually, each - Base Measure, Derived Measure, and Indicator - is composed of all the elements below it on the diagram.)
Key Points:
NOTE: The Measurement Construct is everything in the middle (e.g., between the two gray boxes). The diagram doesn’t define every term relevant to the Model; see Chapter 2 of the PSM book for further details.
A Base Measure value is assigned using a “measurement method,” defined in a data collection procedure, on a given attribute. The method may be objective or subjective (objective methods are preferred where possible).
A Derived Measure value is assigned by performing a “measurement function,” typically a calculation, on one or more Base or Derived Measures.
An Indicator is generated using an “analysis model,” typically an algorithm that combines Base or Derived Measures with associated “decision criteria” to produce a meaningful assessment with respect to an Information Need.
The Information Product includes one or more Indicators with “interpretations” of their significance relative to the Information Need.
Transition:
The ICM table is an example of common measures in use today….
Measurement
Method
Measurement
Method
Operations quantifying an
attribute against a scale
Entities
Attribute
Attribute
Property Relevant to
Information Needs
Part 2

14. Mapping of Information Categories, Measurable Concepts, and Measures (ICM)
The ICM measures are not an “implement all of these” or a “these are all of the possible measures” list
They are measures that have been widely applied and proven to be successful
A different list may be required to address your specific information needs
Main Concept:
This slide show the mapping of PSM Information Categories to Measurable Concepts, and then to Prospective Measures.
Key Points:
NOTE: Part 3 and Appendix A of the PSM book have more details on the Info Needs, Measurable Concepts, and Measures.
The mapping from Information Category to Measurable Concept to Measure is not necessarily two-dimensional, as the table suggests - Measures may apply to more than one Information Category or Measurable Concept. For example, Lines of Code is a measure related to the concept of Physical Size and Stability, but may also be a component of the Productivity measure, which relates to the concept of Process Efficiency.
PSM’s 80+ prospective measures are not all inclusive. These measures have been identified by the PSM Working Group as having been practiced and have been found to be valuable. Measurement users typically add to this collection over time.
Note that the table is a “catalog“ - you don’t have to “buy” everything available. Most projects will not implement all of these measures, and may only implement a few, based on their Information Needs and process capabilities.
Transition:
Now that we have discussed selectcing measures, let’s talking about analysis….
Reference full ICM Table
Part 2

15. Perform Measurement - Key Concepts
This activity provides a systematic method for collecting and transforming measurement data into usable information
Like planning, performing must be flexible to adapt to changing information needs and evolving data
Analysis is the primary task of this activity - it includes estimating, assessing the feasibility of plans, and tracking performance against plans
Main Concept:
In performing measurement, data is used to produce Indicators which provide insight into the program’s issues. This insight helps the team to make better decisions.
Key Points:
PSM defines a process that is both flexible and systematic.
Systematic methods have been defined for analyzing measurement results which can be applied to all measurement.
The analysis process must be flexible enough to accommodate new Information Needs as they arise.
One of the primary tasks in performing measurement is analyzing the Indicators. Analysis may be used to estimate, to evaluate the feasibility of plans, or to monitor performance against project plans.
Transition:
When we’re performing measurement on a project and get some data, there are a number of questions we may ask about the data…
Part 3

16. Perform Measurement
Analyze data to determine what actions need to be taken
Data
Evaluation Measures
Analysis
Results
Information
Questions
New Information Needs
Project and
Enterprise
Context
Analyze
Collect and
Process
Make
Recommendations
Measurement
Plan
Main Concept:
These are the three tasks in the PSM Perform Measurement Activity.
Key Points:
This activity provides a systematic method for collecting and transforming measurement data into usable information
Like planning, performing must be flexible to adapt to changing information needs and evolving data
Collect and Process Data includes collecting, validating, normalizing, aggregating, etc.
Analyze Data includes generating indicators and evaluating alternatives.
Make Recommendations includes making the insight you’ve gained visible to others, along with recommendations based on the information.
Transition:
Let’s talk more about analysis…
Part 3

17. IT Analysis Requirements
The measurement process supports three primary types of information driven analysis activities across the IT program life cycle:
Program estimation and planning
Program feasibility assessments
Acquisition/system performance evaluation
The focus of the measurement process shifts based on:
Which of these activities are currently required
Program life-cycle phase
Program acquisition/development/maintenance strategies
Programmed tasks and activities
Different measurement constructs and analysis techniques are used to generate the required information for each type of analysis
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

18. Integrated Analysis
Integrated analysis combines multiple indicators and focuses on the cause and effect relationships inherent between IT performance parameters
These IT performance parameters map directly to the measurement information categories
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

19. Slides for Practicum 2

20. Measurement Process Model
Core Measurement Activities
Measurement
Plan
New
Issues
Technical and
Management
Processes
Evaluate
User Feedback
Analysis Results
Information
Needs
Improvement
Actions
Analysis Results and
Evaluation Measures
Scope of PSM
Perform
Establish and
Sustain Commitment
Main Concept:
PSM process model (we will expand on this later in the course)
Key Points:
Go over the portions that belong to PSM and how it relates to external processes. Review the flow and interfaces.
Identified information needs feed into the Plan Measurement activity to be used in defining measures
In the Perform Measurement activity, data is turned into indicators that provide insight into the identified information needs.
In Evaluate Measurement, measures and the measurement process are checked to ensure they are providing the necessary insight.
Establish and Sustain Commitment ensures that commitment, resources, and tools are available as needed.
Transition:
Now we will look at the Information Model
Part 1

21. Plan Measurement - Key Concepts
Each project or enterprise is described by a unique set of information needs
Information needs come from objectives and issues (risks, problems, lack of information)
They address questions that need to be answered
The defined and prioritized information needs drive the selection of measures
The measurement definitions and methods are determined by :
The technical and management characteristics of the system (e.g., weapons system vs. MAIS system)
The business and technical processes and capability of the system IT developer(s)/maintainer(s)
Main Concept:
In planning measurement, the information needs to be addressed drive the measures that are selected.
Key Points:
An Information Need is a requirement for insight into issues that could impact the ability to achieve project objectives.
Once a type of measure is selected to address an Information Need, the next step in the planning activity is to look closely at the project processes and determine the specific data elements that may be reasonably collected to obtain that measure.
Transition:
Lets look at the main tasks in the Plan Measurement Activity.
Part 2

22. Plan Measurement
Risk Management
Information
Objectives, Issues
Environment
Improvement Actions
Identify and
Prioritize
Information
Needs
New Information Needs
Select and
Specify
Measures
Main Concept:
These are the main tasks in the Plan Measurement Activity.
Key Points:
We'll be discussing each of the tasks in this lesson.
Note that Plan Measurement is an iterative process. New issues will be identified as the project progresses.
Note the iteration shown by the red arrows. These illustrate that the project processes must be considered in both identifying Information Needs and selecting measures to satisfy the Information Needs.
Once a type of measure is selected to address an Information Need, the next step in planning activity is to look closely at the project processes and determine the specific data elements that may be reasonably collected to obtain that measure.
The results of the Plan Measurement activity are documented in a measurement plan. The "Large Tables" section of your Student Notebook contains a sample measurement plan outline (the measurement plan is discussed later).
Transition:
Let's take a closer look at the first task, Identify and Prioritize Information Needs...
Proposed
Changes
Integrate Into
the Process
Measurement
Plan
The Plan Measurement activity is iterative throughout the life cycle
Part 2

23. Iterative Process
The Plan Measurement activity is iterative throughout the life cycle
Changes will need to occur as information needs change, and as a result of the evaluation of the measurement process and applied measures
Focus on the use of the measures and information products across the stakeholder base - measures should support decision making
Main Concept:
The criteria listed here should be considered in selecting prospective measures. A prospective measure may be a previously-defined measurement construct or an approach to building one.
Key Points:
Criteria:
Alignment with Information Need - Does it answer the right questions?
Effectiveness - How good (timely, accurate, useful) is the measure?
Nature of the Entities to be Measured - The nature of the product or process may determine the unit of measure. For example, reliability of an automobile is best measured in “miles between repair” and aircraft reliability is best measured in “flight hours between repair”. Transaction-based systems are often sized using Function Points.
Cost - Some measures are more expensive than others. For example, for Functional Size and Stability, counting Function Points is more effort intensive and requires more specialized expertise than counting requirements in a requirements database tool.
Life-cycle - You may want to define measures to extend their applicability throughout the life cycle, e.g., extending defect measures to requirements analysis.
External Requirements - You may need to define measures so that they can be used to address higher-level or enterprise Information Needs.
Availability, Integrity, and Usability - Can you get the data, is it accurate, and can it be used for its intended purpose.
Transition:
Let’s talk about the Information Model and how it is used to specify measures.
Part 2

24. Selecting and Specifying Measures
Information requirements unique to the specific program or enterprise (e.g., technical performance, security compliance, portfolio cost performance, etc.), are then combined with the common information requirements to establish a set of program information needs
Initial measurement requirements are assessed for implementation feasibility based on the program’s established management and technical processes
Measures are implemented based on the aligned priorities of the decision information needs
This process iterates across the program life cycle
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

25. Identify Measurement Opportunities
Measurement data can be derived from:
Existing processes, tools, and measurement activities
New data sources developed and integrated to support critical requirements
Most programs have data available from existing tools and measurement activities, including:
Problem report databases
Configuration management systems
Cost, financial management, and schedule control systems
Design tools
Risk management systems
ERP Systems
Main Concept:
In identifying measurement opportunities, we look for existing, cost-effective, minimally disruptive sources for measurement data.
Key Points:
Often, existing tools generate data that can support the selected measures, if we've defined them appropriately.
In looking what exists or what is already planned for a project, consider potential sources for historical data, planning data, and actual performance data; as we will see, all three are useful in the Perform Measurement activity.
Acronym: ERP - Enterprise Resource Planning
Transition:
Once we find sources for data, we need to define the process for collecting and using the data; in other words, we need to specify measurement procedures...
Part 2

26. Document Measurement Plan
Purposes of the plan include:
Define approach for measurement
Define integration of measurement activities with processes
Document measurement procedures
Provide a central source for definitions of measures
Establish a concise working document subject to change
Minimum content
Measurement roles, responsibilities, and communications
Description of Information Needs
Definition of measurement specifications
Main Concept:
The Plan Measurement activity should produce a plan that documents all information needed to implement measurement.
The measurement plan should be concise, but should also provide the details that are necessary for data collection and reporting.
Key Points:
The purpose of the plan is to integrate the measure specifications and data collection and analysis procedures into a unified approach that supports decision makers and provides guidance to data providers, collectors, and analysts.
The measurement plan should be modified as needed throughout the life cycle or over time, adjusting the measures as the information needs change and making improvements to the measures and measurement process as they are identified.
In contractual situations, the measurement plan should be the result of a negotiation between the acquirer and supplier.
This does not have to be a formal deliverable, can be a presentation or part of another plan (e.g., software development plan)
The measurement plan should provide detailed information for data collection, storage, analysis, and delivery.
Note that project Information Needs are documented in the plan; these should be traced to the Measurement Constructs specified to meet them.
Project aggregation structures define how data are to be “rolled up” to show totals for the project. Aggregation structures may be based on product function or structure, phase or activity, responsible enterprise (e.g., subcontractor or vendor), increment, etc. as needed by the project. We'll talk more about aggregation structures in the next lesson.
Transition:
Let’s talk about other considerations for integrating measurement into processes …
Part 2

27. Develop Measurement Procedures
Define the specific operations, tools, and responsibilities for measurement activities
Address both:
Data collection and storage
Data analysis and reporting
Main Concept:
There is more to defining measures than selecting measures and specifying Measurement Constructs. We need also to specify the who, when, where, and how of data collection and analysis.
Key Points:
Measurement procedures must address roles, responsibilities, tools, and procedures.
It's important to define both data collection and storage; well-defined data storage procedures enable historical analysis.
It's important to define both data analysis and reporting; in many unsuccessful measurement approaches, reports are generated, but there is no real analysis. Often, this is because no analysis procedures are defined.
We'll discuss analysis further in the Perform Measurement lesson.
Transition:
These are documented in the measurement plan, along with other information…
Part 2

28. Information Model - High-Level View
Information Need
Can be addressed by many different measures
Information Product
The measures and interpretations
Information Need
Information
Product
Measurable Concept
Describes possible ideas to satisfy information needs
Measurement Construct
Documents the detailed definition of a measure
Measurable
Concept
Measurement
Construct
Entity
Attribute
Entities and Attributes
Specific products and parameters to be measured
Main Concept:
This shows the three-level structure of the Information Model.
Key Points:
At the top, we have an Information Need, which is satisfied by an Information Product. An example would be the need to know whether requirements are growing, satisfied by a requirements volatility indicator with an interpretation of whether volatility is a problem and what we might do about it.
In the middle, we have a Measurable Concept, which is formally expressed through a Measurement Construct. An example of the Measurable Concept would be functional stability, which could be expressed by a measurement construct measuring requirements changes.
At the bottom, we have an entity to be measured; we want to quantify one or more of its attributes. An example of an entity would be a requirements specification and measurable attributes could be the individual requirements or pages.
Transition:
Let’s review what we mean by a Measurement Construct…
Adapted from ISO/IEC/IEEE Measurement Process
Part 2

29. Information Products
Information products are based on the measurement of key program/system attributes at a low level - base measures
Base measures are systematically combined, using consistently defined relationships, to quantify an IT activity or product - the measurement result is then compared against established decision criteria, and presented as a measurement indicator
The measurement indicator conveys the measurement results to the decision maker - it usually compares the measured results to pre- established thresholds that determine the need for action
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

30. Measurement Construct
Information Need
Information Category
Indicator
Analysis Model
Decision Criteria
Base/Derived Measures
For Each Derived Measure:
Measurement Function
Base Measure(s)
For Each Base Measure:
Measurement Method
Type of Method
Scale
Type of Scale
Unit of Measurement
Relevant Entity
Attribute
Main Concept:
These are the details to be specified in a Measurement Construct's specification table.
Key Points:
The construct specification documents:
The Information Need(s) addressed by the measure
The Information Category
The Indicators, analysis model, and decision criteria that make up the final Information Product
Derived measures, which are used to generate the Indicator
Base measures, which are used to generate the Derived Measures
NOTE: For more detailed information, see the PSM book, Chapter 2 and Appendix A.
Transition:
Lets see how the defined measurement constructs are integrated into the Project Processes.
Part 2

31. Information Categories
Backup - additional detail and examples
Information Categories
Schedule and Progress Provides insight into achievement of milestones or events within major project or enterprise activities
Product Quality
Provides insight into the quality of the product
Schedule and Progress
Process Performance Provides insight into the maturity and capability
of the technical and management processes
Resources and Cost
Resources and Cost Provides insight into expenditure of resources, such as people, facilities, and equipment
Size and Stability
Product Quality
Technology Effectiveness Provides insight into suitability, technical performance, and
stability of technologies
Process Performance
Technology Effectiveness
Main Concept:
Project-specific Information Needs can usually be grouped into these Information Categories. Alternatively, you can use the common Information Categories as a starting point for defining project-specific Information Needs.
Key Points:
There are certain types of Information Needs that are common to most projects and enterprises. PSM defines seven Information Categories that correspond to these common Information Needs, and we’ll be using these categories in the process of selecting measures. (Note that while PSM defines seven, you can add to the list if need be.)
Descriptions:
Schedule and Progress - achievement of milestones or events within major project or enterprise activities
Resources and Cost - expenditure of resources, such as people , facilities, and equipment
Size and Stability - changes to product function and structure
Product Quality - the quality of the product
Process Performance - the maturity and capability of the technical and management processes, e.g., CMM level, productivity, defect rates
Technology Effectiveness - the suitability, technical performance, and stability of the technologies
Customer Satisfaction - how well the user’s/customer’s expectations have been met
This list of categories is just a starter set; it's based on experiences on a large variety of projects. If you have an information need that doesn't fit into one of these categories, you can add your own.
NOTE: You can click on the title of this chart to get the definition of every information category, or on one individual category for only its definition.
Transition:
The next slide shows the PSM Information Categories, Measurable Concepts, and Measures...
Customer Satisfaction
Size and Stability Provides insight into changes in product function and structure
Customer Satisfaction Provides insight on how well the user’s / customer’s expectations have been met
Part 2

32. Types of Measures
There are different “types” of information needs. In general these include:
Describing/quantifying IT process and product characteristics
Assessing the accomplishment of programmed activities and the achievement of pre-defined thresholds
Quantifying the changes in outcomes due to a previous action(s)
Assessing the adequacy of the performance of the IT process or product
The different “types” of information needs are supported by different measures and different analysis techniques
Measurement and analysis helps to define objective activity and performance expectations (thresholds) as well as helps to track execution performance against these expectations
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

33. Selecting Prospective Measures
Many different measures are available to satisfy the same information need
Criteria for selecting prospective measures include:
Alignment with Information Need
Measurement effectiveness
Nature of the processes and products to be measured
Cost of data
Life-cycle coverage and applicability
External requirements
Applicability to address multiple information needs
Availability
Integrity (consistency of infrastructure)
Usability
Main Concept:
The criteria listed here should be considered in selecting prospective measures. A prospective measure may be a previously-defined measurement construct or an approach to building one.
Key Points:
Criteria:
Alignment with Information Need - Does it answer the right questions?
Effectiveness - How good (timely, accurate, useful) is the measure?
Nature of the Entities to be Measured - The nature of the product or process may determine the unit of measure. For example, reliability of an automobile is best measured in “miles between repair” and aircraft reliability is best measured in “flight hours between repair”. Transaction-based systems are often sized using Function Points.
Cost - Some measures are more expensive than others. For example, for Functional Size and Stability, counting Function Points is more effort intensive and requires more specialized expertise than counting requirements in a requirements database tool.
Life-cycle - You may want to define measures to extend their applicability throughout the life cycle, e.g., extending defect measures to requirements analysis.
External Requirements - You may need to define measures so that they can be used to address higher-level or enterprise Information Needs.
Availability, Integrity, and Usability - Can you get the data, is it accurate, and can it be used for its intended purpose.
Transition:
Let’s talk about the Information Model and how it is used to specify measures.
Part 2

34. Backup - additional detail and examples
Types of Measures
Characterization Measures
Quantify the attributes of an enterprise, organization, product, project, etc.
Helps to describe and categorize entities based on their characteristics (size, weight, color, quality, functionality, frequencies of occurrence, etc.)
Provide a general context
Activity Measures
Tell you if you are doing what you said you were going to do
Largely relate to schedules and planned accomplishments
Are what most measurement processes initially focus on
Usually structured as expected vs. actual values
Main Concept:
Here are four main types of measures.
Key Points:
The four types of measures we will discuss are Characterization Measures, Activity Measures, Outcome Measures, and Performance Measures. Look at the characteristics of each type.
Transition:
Two are on this slide, with two more on the next slide. We will then look at examples.
Part 1

35. Backup - additional detail and examples
Types of Measures
Outcome Measures
Measure the change in a process, product, or enterprise attribute after an overt action is taken
Address the impact(s) of a previous decision
Encompass explicit exit criteria in the measurement definition
Performance Measures
Measure the change in a process, product, or organizational attribute (outcome) against a defined threshold of required performance
Requires that success factors be defined in measurable terms and linked to measured attributes
Focus on effectiveness, efficiency, quality, and timeliness
Main Concept:
Here are four main types of measures.
Key Points:
The four types of measures we will discuss are Characterization Measures, Activity Measures, Outcome Measures, and Performance Measures. Look at the characteristics of each type.
Transition:
Two are on this slide, with two more on the next slide. We will then look at examples.
Part 1

36. Characterization Measure
Backup - additional detail and examples
Characterization Measure
PSM July 20xx
Project: PSM
Main Concept:
Example of Characterization Measure - Defect Classification
Key Points:
Quantify the attributes of an enterprise, organization, product, project, etc.
Helps to describe and categorize entities based on their characteristics (size, weight, color, quality, frequencies of occurrence, etc.)
Provide a general context
Transition:
Here is another example.
Part 1

37. Backup - additional detail and examples
Activity Measure
PSM July 20xx
Project: PSM
Main Concept:
Example of Activity Measure - Defect status of defects opened vs. closed.
Key Points:
Tell you if you are doing what you said you were going to do
Largely relate to schedules and planned accomplishments
Are what most measurement processes initially focus on
Usually structured as expected vs. actual values
Transition:
Here is another example.
Part 1

38. Backup - additional detail and examples
Outcome Measure
After Process
Improvements
Main Concept:
Example of Outcome Measure - Compares results with baseline or initial values.
Key Points:
Measure the change in a process, product, or enterprise attribute after an overt action is taken
Address the impact(s) of a previous decision
Encompass explicit exit criteria in the measurement definition
Transition:
Here is another example.
Part 1

39. Backup - additional detail and examples
Performance Measure
Main Concept:
Example of Performance Measure
Key Points:
Measure the change in a process, product, or organizational attribute (outcome) against a defined threshold of required performance
Requires that success factors be defined in measurable terms and linked to measured attributes
Focus on effectiveness, efficiency, quality, and timeliness
Transition:
Now let’s look at the difference between project and enterprise measurement
Part 1

40. Key IT Parameters
Program and systems IT performance parameters and associated information needs are highly interrelated
Decisions that result in a change in one area almost always impact another - strong cause and effect relationships exist
The key performance parameters include:
System size and capability
Program resources and cost
Development and maintenance schedule and progress
Product quality
Program decisions must consider these relationships - the measurement and analysis process helps to quantify them
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

41. Key IT Parameters Are Highly Interdependent
Cost
Schedule
Performance
Main Concept:
These parameters are all related and critical to optimal performance.
Key Points:
If we change any of these parameters we need to fully understand the impact to the others. Only then can we optimally manage our performance.
Transition:
What are examples of trade-off decisions that are commonly made.
Product
Quality
Part 1

42. DoD - A Change Driven Environment
The DoD IT environment changes rapidly - making defensible IT decisions is critical to project success
Examples of current challenges include:
Acquisition process changes
Security/risk mandates
Resource constraints
Technology impacts
Life-cycle focus – need to estimate sustainment early
The project environment also changes rapidly
There are different decisions that must be made as the program progresses along its life cycle
There are different risks and problems that must be addressed
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1

43. Process Capability Influences Measurement
A more mature development process provides more opportunities for measurement
Larger selection of base measures - more data availability
Data is better and more consistent - better data integrity
Data is an integral part of the process - usability
Main Concept:
A more mature development process provides more opportunities for measurement.
Key Points:
Can only meaningfully measure things (entities per info model) that are well-defined.
Transition:
Whatever the maturity, when we define measures we need to set targets. Thresholds are set at higher levels of maturity...
Part 2

44. Indicator Targets and Thresholds
Most useful indicators incorporate the notion of measurement targets and thresholds against which actual values can be compared
These help users to understand expected performance and abnormal performance
Targets and thresholds may be derived from:
Plans - Completion criteria
Rules of thumb - Typical values (analogy)
Goals - Descriptive statistics
Delphi methods - Inferential statistics/models
Contract requirements
Main Concept:
Many Indicators compare actual, measured values against some expectation, or baseline, of what these values should be to determine when some action is warranted.
Key Points:
An Indicator Baseline is a set of values for Base or Derived Measures that define the desired or expected level of performance (this is a simple analysis model) against which actual values can be compared.
There are many different sources for Baselines. Some examples:
Plans - Staffing profile from an estimating tool
Rules of thumb - Errors should decrease as testing progresses
Goals - No more than 20% of defects Open throughout the project
Delphi methods - Delphi process to estimate cost for a revision
Contract requirements -
Completion criteria -
Typical values (analogy) - x hours to design a unit
Descriptive statistics -
Inferential statistics/models -
Threshold Value - For reliability, time between failure greater than 500 hours during test
Expectation, Previous Experience - y faults discovered during integration test
Baselines may change over the course of a project or through time for an enterprise, but all previous baselines should be stored in case they are needed for subsequent analysis.
Transition:
Let’s look at some examples of targets and thresholds…..
Part 2

45. IT Measurement & Analysis Requirements
DoDD , DoDI , DAG
Program technical and process performance measures
System effectiveness measures
IT process improvement measures
Information assurance measures
System test and evaluation measures
Cost estimation, earned value, risk management processes
OMB Circular A-11 Part 7, Section 300
Project and enterprise IT portfolio investment performance measures
Cost - schedule - technical - security performance
Objective project and portfolio risk assessment
Performance transparency - IT Dashboard
Exhibit 53 - Exhibit 300 reports
Main Concepts:
Measurement helps managers succeed.
Key Points:
Measurement helps the managers track and control projects and the enterprise on a daily basis.
When used correctly, measurement can be a proactive tool to help head off potential problems (or at least minimize the impact of problems).
Measurement also generates the data that can help plan future projects.
The PSM process allows managers to quantify risks and assess the impacts of potential problems on project cost, schedule, quality, and performance objectives. Measurement also supports development of alternative solutions, and selection of the best approach to avoid or correct potential problems.
A measurement approach that is directly related to project objectives and issues encourages buy-in and commitment to collect data. It provides real data to accurately and objectively monitor the impact of management decisions and project changes.
Transition:
Let's look at today’s environment
Part 1