NSA MOE-PIR Outcome Measures of Effectiveness (MOEs) and their Measure During a Post Implementation Review (PIR) Leonard Sadauskas OASD(CIO) CP&O

NSA MOE-PIR Scope and Objective This presentation deals with two elements of performance and results-based management of IT and National Security Systems (NSS) –Prescribing IT outcomes performance measures; defined as measures of effectiveness, MOEs –Measuring the MOEs during a Post-Implementation Review The objective is to answer who, what, why and when for MOEs and their measure Noel Dickover will introduce you to additional resources of MOE/PIR information at the IT-Community of Interest:

NSA MOE-PIR MOEs Defined MOEs are independent of any solution and do not specify performance or criteria. The MOE is the criterion by which solutions will be judged MOEs are a measure of operational success that must be closely related to the objective of the mission or operation being evaluated. For example, the number of enemy submarines sunk or enemy tanks destroyed may be satisfactory MOEs if the objective is to destroy such weapons systems. A meaningful MOE must be quantifiable and a measure to what degree the real objective is achieved.

NSA MOE-PIR PIR Defined A PIR is an analysis of an investment or acquired system, operating in its intended environment, using data collected from various sources to answer the question: Did we get what we needed, and if not what to do about it? A PIR is a measurement of the outcome measures of effectiveness (MOEs) assigned to a materiel solution by the capability/mission owner

NSA MOE-PIR Statutory Requirements Sec 5123 of the Clinger-Cohen Act: (Performance and Results-based Management) Establish goals for improving Agency efficiency and effectiveness Provide annual status of goals to Congress Prescribe IT/NSS outcomes performance measures (MOEs) Benchmark against comparable processes and organizations Measure how well the outcomes support programs of the Agency (PIR) Meet fundamental criteria: –Core/priority mission functions for performance by Government –No alternative private sector or government source (e-GOV) Conduct BPR/Change management before investing in IT/NSS Ensure information assurance + Annual Appropriations Legislation implemented by DoD CIO memoranda

NSA MOE-PIR Regulatory Requirements JCIDS Manual assigns the respective FCBs and Domains responsibility for articulating capability attributes, metrics and measures DODI –Table E3.T1 cites information required for the Full-Rate Production decision, including the Post-Deployment Performance Review citing GPRA and 40USC (CCA) –Table E4.T1 “establish outcome-based performance measures linked to strategic goals”. OMB Circular A-130 in section under “Evaluation” –Conduct post-implementation reviews of IT to validate estimated benefits and costs OMB Circular A-11 Exhibit 300, Table 1C requires submission of: –Performance Goals and Measures. Strategic goal(s) supported; Existing baseline; Planned and actual improvements and performance metrics

NSA MOE-PIR CCA & E-GOV GPRABalanced Scorecard Baldrige Quality Criteria Six Sigma Mission & Business Results Customer Results Processes & Activities Human Capital Technology Measurement Area Drivers

NSA MOE-PIR MOEs and PIR in the Life-cycle The following slides provide three views of MOEs and the PIR –As an OODA Loop –In the investment life-cycle –In the JCIDS life-cycle

NSA MOE-PIR OODA Loop in IT Investments Observe Orient Decide Act Feedback DOTMLPF Experimentation AoA Functional Area & Needs Analysis Acquire Test Field JROC AMS PIR Ground Truth

NSA MOE-PIR IT/NSS Investment Life Cycle PIR ICD (MOEs) CDD (KPPs) CPD Deployment IOC-FOC OT&E JCIDS Analysis FAA*FNA*FSA An Evolutionary Process

NSA MOE-PIR MOE/PIR in the JCIDS Process Existing and Proposed JCIDS Processes for Satisfying CCA JCD Joint Capabilities Document DCR DOTMLPF Change Recommendation ICD Initial Capabilities Document CDD Capability Development Document CPD Capability Production Document PIRD Post Implementation Review Document CPDPIRD Functional Area Analysis Functional Needs Analysis Alternative N Alternative 2 DCR Functional Solution Analysis CDD CPD Alternative 1 Joint Operations Concepts DOTMLPF Analysis (non-materiel Approaches) JCD ICD Joint Operating Concepts Joint Functional Concepts Joint Integrating Concepts Functional Area Analysis Functional Needs Analysis Ideas for Materiel Approaches Analysis of Materiel/ Non-Materiel Approaches Alternative N Alternative 2 DCR Functional Solution Analysis DOD Strategic Guidance CDD CPD Alternative 1 Joint Operations Concepts DOTMLPF Analysis JCD ICD Integrated Architectures Post Independent Analysis 1. Facilitates development of measures of effectiveness (MOEs) 11. Contains MOEs 7. BPR Analysis 8. No private or other government organization better performs function 5. Core function need to be performed by government 4. Identifies functional area metrics and submits MOEs 3. Output is capabilities and their associated tasks & attributes 12. IT Architecture 14. PROPOSED Assessment of Outcomes 2. Provides fundamental tasks and measures required for CBA 6. Prioritized COCOM Needs 9. May satisfy AIS AoA 13. MISSING Measurement of the MOEs 10. Initiates non- materiel BPR Execution

NSA MOE-PIR Developing the MOEs MOEs are derived from Functional Concept capability attributes, measures and metrics –Describe the outcome expectations of the Functional Sponsor New capability Improving existing capability CIO provides leadership in Net-centric FCB and supports development of related MOEs

NSA MOE-PIR Top Level Capabilities and Metric Summary CapabilityAttributesMeasuresMetrics Current (Baseline) Value Targeted Threshold Value Targeted Objective Value Capability #1 Attribute #1 Measures...Metric 1 Value for Metric 1 Attribute #2 Measures...Metric 1, 3 Value for M1 Value for M3 Value for M1 Value for M3 Capability #2 Attribute #2 Measures...Metric... Value for Metrics... Attribute #4 Measures...Metric... Value for Metrics... Capability #2 Attribute #1 Measures...Metric... Value for Metrics... Attribute #3 Measures...Metric... Value for Metrics... Attribute #4 Measures...Metric... Value for Metrics... Attribute #5 Measures...Metric... Value for Metrics... From Functional Concept Capabilities <> Attributes Matrix and Service Capabilities Documents Determined by Team Qualitative / Quantified From Functional Concept Attribute Measures <> Metric Matrices

NSA MOE-PIR Joint C2 Example – From FC v1.0 Monitor & collect data Understand the situation C2 CAPABILITIES ATTRIBUTES Superior Decision Making SAMPLEMEASURES SAMPLEMEASURES Timeliness of Decision SAMPLEMETRICS SAMPLEMETRICS Extent to which currency of a decision is appropriate to the mission

NSA MOE-PIR Joint C2 Example – Values TBD METRIC CURRENT (BASELINE) VALUE –Data from past assessments May need to conduct an assessment –Likely to be quantification of a qualitative questionnaire Respondents consider currency of present process to be appropriate only 40% of the time TARGETED THRESHOLD & OBJECTIVE VALUE –Needs consensus and chop by Sponsor and FCB Chair Extent to which currency of a decision is appropriate to the mission

NSA MOE-PIR Other MOE Examples Customer Satisfaction: –80% of respondents assess category ABC as above threshold Economy: –Return on Investment greater than 300% Efficiency: –Maintenance manpower reduction of 50% Effectiveness: –Reduce IED events by 50% –Increase available aircraft employment to 80% –Decrease uncertainty of XYZ assessments by 30 % –Enable rapid detection and classification of short burst signals –Reduce manned aircraft requirements by 50%

NSA MOE-PIR NOT MOEs Programmatic measures –Cost, Schedule Solution dependent KPPs –Solution technical parameters –System outputs Note Some MOEs are also KPPs and are tested during OT&E

NSA MOE-PIR Planning and Implementing the PIR Start planning during MOE development –Affordability of proposed measures Gather as-is data for comparison Evolve plan with final at FRPD Schedule the PIR –Make maximum use of scheduled events Assemble and Charter the PIR Team –Functional, Technical, CIO, Oversight Conduct the PIR Conduct the Analysis Prepare and Sign Report –Provide Recommendations for future increments FUNCTIONAL SPONSOR Program Manager Support Functional Sponsor

NSA MOE-PIR Post Implementation Review (PIR) in the Acquisition Life Cycle ICD MS A MS B CPD DT&E OT&E MS C FRPD PIR Plan TEMP MISSION OUTCOME PERFORMANCE MEASURES PLAN FOT&E Platform Readiness Assessments CC Exercise results User Satisfaction Surveys Annual CFO Report Input Mission Readiness Assessments ROI Computation PIR SEP Integration & Test Contract IOC FCB/PSA MOE PIR Build FCB: Functional Capabilities Board ICD: Initial Capabilities Document CDD: Capability Development Document CPD: Capability Production Document CDD FOC

NSA MOE-PIR Common Problems with PIR Multiple organizations with no common measurement standards Lack of documentation in ICD and/or business case Lack of Functional Sponsor ownership of PIR Sensitivities of participating organizations Doing a PIR without using the results

NSA MOE-PIR Guidance and Practitioner’s Resources Ch 7 of DAG References and Examples in IT-CoP One-to-many discussions on IT-CoP-L More during Noel’s Presentation

NSA MOE-PIR Questions?

NSA MOE-PIR Acronyms AMS: Acquisition Management System AoA: Analysis of Alternatives BPR: Business Process Reengineering C2: Command & Control CCA: Clinger-Cohen Act of 1996, formerly the Information Technology Management Reform Act of 1996 CDD: Capability Development Document CIO: Chief Information Officer CPD: Capability Production Cocument DCR: DOTMLPF Change Request DOTMLPF: doctrine, organization. Training, materiel, leadership and education, personnel, facilities E-Gov: E-Government Act of 2002 FAA: Functional Area Analysis FCB: Functional Control Board FNA: Functional Needs Analysis FOC: Full Operational Capability FSA: Functional Solution Analysis GPRA: Government Performance and Results Act 1993 ICD: Initial Capabilities Document IOC: Initial Operational Capability IT: Information Technology JCD: Joint Capabilities Document JCIDS: Joint Capability Integration and Development System JROC: Joint Requirements Oversight Council KPP: Key Performance Parameter MOE: Measure of Effectiveness NSS: National Security System OMB: Office of Management and Budget OODA: Observe, Orient, Decide, Act OT&E: Operational Test & Evaluation PIR: Post Implementation Review