1. Peace Operations Policy Program, School of Public Policy POPP Value Focused Metrics 26 ISMOR D.F. Davis A.D. Caldwell ddavis@gmu.edu andrew.caldwell.uk@osd.mil

2. Peace Operations Policy Program, School of Public Policy POPP VFM Value Focused Metrics is a term stolen from Keeney’s Value Focused Thinking. This use is not unique and a simple Google search brings out 99 other uses. Most of these are in the context of “Agile Human Resources,” “Agile Systems Design.” and “Project Management.” Similar uses have been around since the early 1900s, but mostly since the maturing of Decision Science in the 1960s. Value Focused Thinking by Ralph Keeney came out in 1992.

3. Peace Operations Policy Program, School of Public Policy POPP Insight “The purpose of computing is insight, not numbers.” Richard W. Hamming while at IBM labs “The purpose of analysis is insight, not numbers.” Wilbur Payne while at TRADOC Analysis Command “Our purpose is insight, not metrics.” Dave Davis while at ISMOR

4. Peace Operations Policy Program, School of Public Policy POPP RTO-TR-081 Page 5-1 NATO Code of Best Practices For Command and Control Assessment Metrics should flow from the process, not be separated from it.

5. Peace Operations Policy Program, School of Public Policy POPP The Code of Best Practice has adopted the following five levels of MoM (Measures of Merit): Measures of Policy Effectiveness (MoPE), which focus on policy and societal outcomes; Measures of Force Effectiveness (MoFE), which focus on how a force performs its mission or the degree to which it meets its objectives; Measures of C2 Effectiveness (MoCE), which focus on the impact of C2 systems within the operational context; Measures of Performance (MoP), which focus on internal system structure, characteristics and behaviour; and Dimensional Parameters (DP), which focus on the properties or characteristics inherent in the physical C2 systems. RTO-TR-081 Page 5-2 COBP

6. Peace Operations Policy Program, School of Public Policy POPP RTO-TR-081 Page 5-3 COBP This figure implies that metrics are nested. I’m not sure this is always the case.

7. Peace Operations Policy Program, School of Public Policy POPP MPICE Sectors Goals Drivers/Capacities Indicators Measures

8. Peace Operations Policy Program, School of Public Policy POPP

9. Peace Operations Policy Program, School of Public Policy POPP

10. Peace Operations Policy Program, School of Public Policy POPP R ICHARDSON, H, 2004, MOE AND EBO IN THE H EADQUARTERS ARRC, IN T HE C ORNWALLIS C ROUP IX: A NALYSIS FOR S TABILIZATION AND C OUNTER -T ERRORIST O PERATIONS, T HE C ANADIAN P EACE K EEPING P RESS ( HTTP :// WWW.T HE C ORNWALLIS G ROUP. ORG ), P83 HTTP :// WWW.T HE C ORNWALLIS G ROUP. ORG

11. Peace Operations Policy Program, School of Public Policy POPP R ICHARDSON, H, 2004, MOE AND EBO IN THE H EADQUARTERS ARRC, IN T HE C ORNWALLIS C ROUP IX: A NALYSIS FOR S TABILIZATION AND C OUNTER -T ERRORIST O PERATIONS, T HE C ANADIAN P EACE K EEPING P RESS ( HTTP :// WWW.T HE C ORNWALLIS G ROUP. ORG ), P85 HTTP :// WWW.T HE C ORNWALLIS G ROUP. ORG

12. Peace Operations Policy Program, School of Public Policy POPP R ICHARDSON, H, 2004, MOE AND EBO IN THE H EADQUARTERS ARRC, IN T HE C ORNWALLIS C ROUP IX: A NALYSIS FOR S TABILIZATION AND C OUNTER -T ERRORIST O PERATIONS, T HE C ANADIAN P EACE K EEPING P RESS ( HTTP :// WWW.T HE C ORNWALLIS G ROUP. ORG ), P87 HTTP :// WWW.T HE C ORNWALLIS G ROUP. ORG

13. Peace Operations Policy Program, School of Public Policy POPP Chernoff Faces

14. Peace Operations Policy Program, School of Public Policy POPP 1)Confusion between objectives, activities, inputs, outputs, outcomes, ways, means, ends,... 2)What influences what, 3)Are metrics dependent on the objectives or are objectives dependent on the metrics? 4)Roll up the metrics or display for mental integration 5)Whose toy should we use? Several Issues

15. Peace Operations Policy Program, School of Public Policy POPP Bumper Sticker Metrics should be Organic to the Plan

16. Peace Operations Policy Program, School of Public Policy POPP Value Focused Metrics Metrics should be organic to the objectives and flow from the plan, not come from separate lists or be tacked on at the end. Value Focused Metrics (VFM) says: First identify your objectives, then determine how each Fundamental Objective can be served by a network or Means Objectives. This is the ‘Model’ of the overall intervention/plan or mission. Once the structure and content of the Model is agreed, then identify what the interveners can do or control – what activities, projects and programs need to be put in place, and what Means Objectives are these expected to influence. Finally, when the Model is enhanced by the activities, then and only then determine what Metrics should be associate with both Fundamental and Means Objectives. Keep track of resources, but they are controls on the activities, not metrics.

17. Peace Operations Policy Program, School of Public Policy POPP Some definitions from Keeney (and some from Davis): 1)Objective (Keeney, p34): “..statement of something that one desires to achieve.” a)Context b)Object c)Preference 2)Goal (K60) (An objective with values, standards or thresholds.) 3)Fundamental Objectives: “Just Because” 4)Means Objectives: “In order to accomplish X” 5)Attribute (K100): “The degree to which an objective is achieved..” 6)Metric (Davis): The definition of an attribute 7)Measurement (Davis): The collected value of a metric. Definitions

18. Peace Operations Policy Program, School of Public Policy POPP What the METRICS will mean Measurements associated with Metrics on the Fundamental Objectives indicate overall Outcomes, or mission achievement. Are we accomplishing the overall effects that we are there to create? Measurements associated with Metrics on the Means Objectives indicate how well the plan is being accomplished. Are we using the right ways and means to accomplish our goals? Metrics on Fundamental Objectives are for Mission Achievement. Metrics on Means Objectives are for Managerial Efficiency. BOTH ARE VALID AND NECESSARY BUT they ARE NOT THE SAME THING!!!!

19. Peace Operations Policy Program, School of Public Policy POPP The Process The modeling process starts with developing a Fundamental Objectives Hierarchy (FOH) and a Means Ends Objectives Network (MEON). A MEON is a model of causality, it is meant to show how a Fundamental Objective (FO) can be met. To do so, there are several requirements: 1)A list of domain elements must be collected – what are the things in this domain that impact or may impact the FO? For the intervention domain we are using the ETL/MPICE/CMPO. This yields CONTENT. 2)These domain elements must be arrayed in a network showing both causal and correlation linkages. We are doing this using Microsoft Belief Networks (a freeware from MS that implements Bayesian Networks). This process provides us with STRUCTURE. 3)Once the CONTENT has been arrayed in the STRUCTURE then the theories of change must be added. In the case of Bayes Nets, this theory is one of probabilities. This then gives us VALUE. 4)Drivers/Activities/Programs/Projects can then be added to represent what the interveners can actually CONTROL. Activities may require RESOURCES. 5)Many, but not necessarily all, objectives can have attributes, or METRICS attached.

20. Peace Operations Policy Program, School of Public Policy POPP Sources of the Elements Catalogs – MPICE Workshops and expert opinion Affinity Lists (Parnell) Old models “The Plan”

21. Peace Operations Policy Program, School of Public Policy POPP What's a what For each element in your list, ask first: 1)Why is this important? a)“In order to do this or that” – Add to list of Means Objectives b)“Just because, or That is the reason we are here” – Add to list of Fundamental Objectives c)“Its something that we do” – add to list of potential activities 2)If a Means objective, what does it influence? (potential link) 3)For every objective, fundamental or means: How would we know that this objective has been accomplished? Candidate attributes or metrics. Then, of course, ask: What have we forgotten?

22. Peace Operations Policy Program, School of Public Policy POPP Fundamental Objective Hierarchy Means Ends Objective Network Controls, Activities, Programs, Projects The System Metrics Resources

23. Peace Operations Policy Program, School of Public Policy POPP Structure - Basic The Yellow node is the Strategic (Fundamental) Objective. The Gray nodes are the Means Objectives linked to represent how we think they support each other.

24. Peace Operations Policy Program, School of Public Policy POPP Structure - Influences The Green nodes are activities, things we can actively control

25. Peace Operations Policy Program, School of Public Policy POPP Structure - Metrics The Blue nodes are the metrics – the metrics depend on the objectives and do not form their own, independent system.

26. Peace Operations Policy Program, School of Public Policy POPP Value Value depends on classifying the objective’s possible states. In this case Needs can be in either of the states Provided, Marginal, or Significant need persist. Value is completed by assessing the local theories (probabilities) at each node. For example, the probability that Needs will be Provide For GIVEN that Shelter is Sufficient AND Education is Available AND Access to Water is Sufficient is.897. Whereas the probability that Needs will be Provided For GIVEN that Shelter is Available Below Standards AND Education is Marginal AND Access to Water is Marginal is only.526. These theories are based on input assessments by experts.

27. Peace Operations Policy Program, School of Public Policy POPP Conclusion Metrics should be Organic to the Plan

28. Peace Operations Policy Program, School of Public Policy POPP Questions?