1. NSA MOE-PIR 20051 Outcome Measures of Effectiveness (MOEs) and their Measure During a Post Implementation Review (PIR) Leonard Sadauskas OASD(CIO) CP&O Leonard.Sadauskas.ctr@osd.mil

2. NSA MOE-PIR 20052 Scope and Objective This presentation deals with two elements of performance and results-based management of IT and National Security Systems (NSS) –Prescribing IT outcomes performance measures; defined as measures of effectiveness, MOEs –Measuring the MOEs during a Post-Implementation Review The objective is to answer who, what, why and when for MOEs and their measure Noel Dickover will introduce you to additional resources of MOE/PIR information at the IT-Community of Interest: http://acc.dau.mil/cca http://acc.dau.mil/cca

3. NSA MOE-PIR 20053 MOEs Defined MOEs are independent of any solution and do not specify performance or criteria. The MOE is the criterion by which solutions will be judged MOEs are a measure of operational success that must be closely related to the objective of the mission or operation being evaluated. For example, the number of enemy submarines sunk or enemy tanks destroyed may be satisfactory MOEs if the objective is to destroy such weapons systems. A meaningful MOE must be quantifiable and a measure to what degree the real objective is achieved.

4. NSA MOE-PIR 20054 PIR Defined A PIR is an analysis of an investment or acquired system, operating in its intended environment, using data collected from various sources to answer the question: Did we get what we needed, and if not what to do about it? A PIR is a measurement of the outcome measures of effectiveness (MOEs) assigned to a materiel solution by the capability/mission owner

5. NSA MOE-PIR 20055 Statutory Requirements Sec 5123 of the Clinger-Cohen Act: (Performance and Results-based Management) Establish goals for improving Agency efficiency and effectiveness Provide annual status of goals to Congress Prescribe IT/NSS outcomes performance measures (MOEs) Benchmark against comparable processes and organizations Measure how well the outcomes support programs of the Agency (PIR) Meet fundamental criteria: –Core/priority mission functions for performance by Government –No alternative private sector or government source (e-GOV) Conduct BPR/Change management before investing in IT/NSS Ensure information assurance + Annual Appropriations Legislation implemented by DoD CIO memoranda

6. NSA MOE-PIR 20056 Regulatory Requirements JCIDS Manual assigns the respective FCBs and Domains responsibility for articulating capability attributes, metrics and measures DODI 5000.2 –Table E3.T1 cites information required for the Full-Rate Production decision, including the Post-Deployment Performance Review citing GPRA and 40USC 11313 (CCA) –Table E4.T1 “establish outcome-based performance measures linked to strategic goals”. OMB Circular A-130 in section under “Evaluation” –Conduct post-implementation reviews of IT to validate estimated benefits and costs OMB Circular A-11 Exhibit 300, Table 1C requires submission of: –Performance Goals and Measures. Strategic goal(s) supported; Existing baseline; Planned and actual improvements and performance metrics

7. NSA MOE-PIR 20057 CCA & E-GOV GPRABalanced Scorecard Baldrige Quality Criteria Six Sigma Mission & Business Results Customer Results Processes & Activities Human Capital Technology Measurement Area Drivers

8. NSA MOE-PIR 20058 MOEs and PIR in the Life-cycle The following slides provide three views of MOEs and the PIR –As an OODA Loop –In the investment life-cycle –In the JCIDS life-cycle

9. NSA MOE-PIR 20059 OODA Loop in IT Investments Observe Orient Decide Act Feedback DOTMLPF Experimentation AoA Functional Area & Needs Analysis Acquire Test Field JROC AMS PIR Ground Truth

10. NSA MOE-PIR 200510 IT/NSS Investment Life Cycle PIR ICD (MOEs) CDD (KPPs) CPD Deployment IOC-FOC OT&E JCIDS Analysis FAA*FNA*FSA An Evolutionary Process

11. NSA MOE-PIR 200511 MOE/PIR in the JCIDS Process Existing and Proposed JCIDS Processes for Satisfying CCA JCD Joint Capabilities Document DCR DOTMLPF Change Recommendation ICD Initial Capabilities Document CDD Capability Development Document CPD Capability Production Document PIRD Post Implementation Review Document CPDPIRD Functional Area Analysis Functional Needs Analysis Alternative N Alternative 2 DCR Functional Solution Analysis CDD CPD Alternative 1 Joint Operations Concepts DOTMLPF Analysis (non-materiel Approaches) JCD ICD Joint Operating Concepts Joint Functional Concepts Joint Integrating Concepts Functional Area Analysis Functional Needs Analysis Ideas for Materiel Approaches Analysis of Materiel/ Non-Materiel Approaches Alternative N Alternative 2 DCR Functional Solution Analysis DOD Strategic Guidance CDD CPD Alternative 1 Joint Operations Concepts DOTMLPF Analysis JCD ICD Integrated Architectures Post Independent Analysis 1. Facilitates development of measures of effectiveness (MOEs) 11. Contains MOEs 7. BPR Analysis 8. No private or other government organization better performs function 5. Core function need to be performed by government 4. Identifies functional area metrics and submits MOEs 3. Output is capabilities and their associated tasks & attributes 12. IT Architecture 14. PROPOSED Assessment of Outcomes 2. Provides fundamental tasks and measures required for CBA 6. Prioritized COCOM Needs 9. May satisfy AIS AoA 13. MISSING Measurement of the MOEs 10. Initiates non- materiel BPR Execution

12. NSA MOE-PIR 200512 Developing the MOEs MOEs are derived from Functional Concept capability attributes, measures and metrics –Describe the outcome expectations of the Functional Sponsor New capability Improving existing capability CIO provides leadership in Net-centric FCB and supports development of related MOEs

13. NSA MOE-PIR 200513 Top Level Capabilities and Metric Summary CapabilityAttributesMeasuresMetrics Current (Baseline) Value Targeted Threshold Value Targeted Objective Value Capability #1 Attribute #1 Measures...Metric 1 Value for Metric 1 Attribute #2 Measures...Metric 1, 3 Value for M1 Value for M3 Value for M1 Value for M3 Capability #2 Attribute #2 Measures...Metric... Value for Metrics... Attribute #4 Measures...Metric... Value for Metrics... Capability #2 Attribute #1 Measures...Metric... Value for Metrics... Attribute #3 Measures...Metric... Value for Metrics... Attribute #4 Measures...Metric... Value for Metrics... Attribute #5 Measures...Metric... Value for Metrics... From Functional Concept Capabilities <> Attributes Matrix and Service Capabilities Documents Determined by Team Qualitative / Quantified From Functional Concept Attribute Measures <> Metric Matrices

14. NSA MOE-PIR 200514 Joint C2 Example – From FC v1.0 Monitor & collect data Understand the situation C2 CAPABILITIES ATTRIBUTES Superior Decision Making SAMPLEMEASURES SAMPLEMEASURES Timeliness of Decision SAMPLEMETRICS SAMPLEMETRICS Extent to which currency of a decision is appropriate to the mission

15. NSA MOE-PIR 200515 Joint C2 Example – Values TBD METRIC CURRENT (BASELINE) VALUE –Data from past assessments May need to conduct an assessment –Likely to be quantification of a qualitative questionnaire Respondents consider currency of present process to be appropriate only 40% of the time TARGETED THRESHOLD & OBJECTIVE VALUE –Needs consensus and chop by Sponsor and FCB Chair Extent to which currency of a decision is appropriate to the mission

16. NSA MOE-PIR 200516 Other MOE Examples Customer Satisfaction: –80% of respondents assess category ABC as above threshold Economy: –Return on Investment greater than 300% Efficiency: –Maintenance manpower reduction of 50% Effectiveness: –Reduce IED events by 50% –Increase available aircraft employment to 80% –Decrease uncertainty of XYZ assessments by 30 % –Enable rapid detection and classification of short burst signals –Reduce manned aircraft requirements by 50%

17. NSA MOE-PIR 200517 NOT MOEs Programmatic measures –Cost, Schedule Solution dependent KPPs –Solution technical parameters –System outputs Note Some MOEs are also KPPs and are tested during OT&E

18. NSA MOE-PIR 200518 Planning and Implementing the PIR Start planning during MOE development –Affordability of proposed measures Gather as-is data for comparison Evolve plan with final at FRPD Schedule the PIR –Make maximum use of scheduled events Assemble and Charter the PIR Team –Functional, Technical, CIO, Oversight Conduct the PIR Conduct the Analysis Prepare and Sign Report –Provide Recommendations for future increments FUNCTIONAL SPONSOR Program Manager Support Functional Sponsor

19. NSA MOE-PIR 200519 Post Implementation Review (PIR) in the Acquisition Life Cycle ICD MS A MS B CPD DT&E OT&E MS C FRPD PIR Plan TEMP MISSION OUTCOME PERFORMANCE MEASURES PLAN FOT&E Platform Readiness Assessments CC Exercise results User Satisfaction Surveys Annual CFO Report Input Mission Readiness Assessments ROI Computation PIR SEP Integration & Test Contract IOC FCB/PSA MOE PIR Build FCB: Functional Capabilities Board ICD: Initial Capabilities Document CDD: Capability Development Document CPD: Capability Production Document CDD FOC

20. NSA MOE-PIR 200520 Common Problems with PIR Multiple organizations with no common measurement standards Lack of documentation in ICD and/or business case Lack of Functional Sponsor ownership of PIR Sensitivities of participating organizations Doing a PIR without using the results

21. NSA MOE-PIR 200521 Guidance and Practitioner’s Resources Ch 7 of DAG References and Examples in IT-CoP One-to-many discussions on IT-CoP-L More during Noel’s Presentation

22. NSA MOE-PIR 200522 Questions?

23. NSA MOE-PIR 200523 Acronyms AMS: Acquisition Management System AoA: Analysis of Alternatives BPR: Business Process Reengineering C2: Command & Control CCA: Clinger-Cohen Act of 1996, formerly the Information Technology Management Reform Act of 1996 CDD: Capability Development Document CIO: Chief Information Officer CPD: Capability Production Cocument DCR: DOTMLPF Change Request DOTMLPF: doctrine, organization. Training, materiel, leadership and education, personnel, facilities E-Gov: E-Government Act of 2002 FAA: Functional Area Analysis FCB: Functional Control Board FNA: Functional Needs Analysis FOC: Full Operational Capability FSA: Functional Solution Analysis GPRA: Government Performance and Results Act 1993 ICD: Initial Capabilities Document IOC: Initial Operational Capability IT: Information Technology JCD: Joint Capabilities Document JCIDS: Joint Capability Integration and Development System JROC: Joint Requirements Oversight Council KPP: Key Performance Parameter MOE: Measure of Effectiveness NSS: National Security System OMB: Office of Management and Budget OODA: Observe, Orient, Decide, Act OT&E: Operational Test & Evaluation PIR: Post Implementation Review