1. Measures, Metrics, and Systems-of-Systems Bridging a Gap Between Academic and DoD Systems Engineering Terminology Dr. Thomas Holland, Ph.D. Naval Surface Warfare Center Dahlgren, VA NSWCDD-PN-14-00163 Public Release Statement A: Approved for Public Release

2. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Overview  Measure or Metric?  Systems Perspective and the DoD  MOEs & MOPs  Systems vs. Systems-of-Systems  Military Definitions of SoS  In Conclusion 2 Public Release Statement A: Approved for Public Release

3. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” 3 Public Release Statement A: Approved for Public Release

4. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” 4 Public Release Statement A: Approved for Public Release

5. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” 5 Public Release Statement A: Approved for Public Release

6. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” 6 Public Release Statement A: Approved for Public Release

7. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” 7 Public Release Statement A: Approved for Public Release

8. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? Measure: a dimension, quantity, or capacity (degree), ascertained by comparison to an accepted standard. length in inches weight in pounds temperature in degrees Celsius (Oxford Dictionary, 2013) 8 Public Release Statement A: Approved for Public Release

9. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Measure or Metric? Metric: a technical system or standard of measurement; a set of figures or statistics that measure results. Productivity: number of sales calls an employee makes per day. Efficiency: probability of correct data entry. (Oxford Dictionary, 2013) 9 Public Release Statement A: Approved for Public Release

10. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems Perspective & the DoD  In SE, “metric” refers to performance. “The SoS metrics are collected and analyzed as part of analyses to assess whether the SoS is making progress towards objectives.” (OAD, 2008) Despite these positive indications, the United States has not declared victory against terrorism nor has the U.S. government satisfied world or domestic opinion with explanations of forward progress in the war. Metrics must be developed to give meaning to our military actions.” (Walker, 2005) 10 Public Release Statement A: Approved for Public Release

11. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems Perspective & the DoD  We can measure many things.  But there are certain measures and collections of measures that are important within the context of our interest.  The right metric helps us answer our specific questions. 11 Public Release Statement A: Approved for Public Release

12. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems Perspective & the DoD Metric: a set of measures or methods that ascertain the progress a system is making toward achieving its goal.  Did you know, “Metric” is not defined in the DAU Glossary? 12 Public Release Statement A: Approved for Public Release

13. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 MOEs and MOPs Measure of Effectiveness (MOE): “The data used to measure the military effect (mission accomplishment) that comes from the use of the system in its expected environment. That environment includes the system under test and all interrelated systems, that is, the planned or expected environment in terms of weapons, sensors, command and control, and platforms, as appropriate, needed to accomplish an end-to-end mission in combat.” (Defense Acquisition University, 2013) 13 Public Release Statement A: Approved for Public Release

14. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 MOEs and MOPs Measure of Performance (MOP): “System-particular performance parameters such as speed, payload, range, time-on-station, frequency, or other distinctly quantifiable performance features. Several MOPs may be related to the achievement of a particular Measure of Effectiveness (MOE).” (Defense Acquisition University, 2013) 14 Public Release Statement A: Approved for Public Release

15. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 MOEs and MOPs Public Release Statement A: Approved for Public Release 15

16. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 16 Public Release Statement A: Approved for Public Release

17. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 17 Public Release Statement A: Approved for Public Release

18. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 18 Public Release Statement A: Approved for Public Release

19. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 19 Public Release Statement A: Approved for Public Release

20. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 20 Public Release Statement A: Approved for Public Release

21. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 21 Public Release Statement A: Approved for Public Release

22. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems  System:any interacting or interdependent group of entities that forms a unified and purposeful whole.  System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) 22 Public Release Statement A: Approved for Public Release

23. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems Public Release Statement A: Approved for Public Release 23 Traditional Monolithic Systems Systems-of-Systems Centralized ControlIndependent Decisions Brittle (linear)Adaptive (complex) Behaviors SpecifiedBehaviors Emerge Closed System ConstraintsUnbounded Constraints Hierarchical Structures (Components)Autonomous Decisions (Constituents) Components may have Global VisibilityConstituents have Local Visibility Tight Coupling between ComponentsDynamic Interactions between Constituents Components are IntegratedConstituents are Interoperable

24. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Military Definitions of SoS System: A functionally, physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole. 24 Public Release Statement A: Approved for Public Release

25. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Military Definitions of SoS System-of-systems: a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities. Both individual systems and SoS conform to the accepted definition of a system in that each consists of parts, relationships, and a whole that is greater than the sum of the parts; however, although an SoS is a system, not all systems are SoS. Public Release Statement A: Approved for Public Release 25

26. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Systems vs. Systems-of-Systems Public Release Statement A: Approved for Public Release 26 SoS: Context not entirely in original systems’ contexts SoS may exclude some of the original context FOS: Shared Context of systems operating independently Context for A Context for B SoS makes some of the original context internal

27. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Military Definitions of SoS Academic System: any interacting or interdependent group of entities that forms a unified and purposeful whole. DoD System: a functionally, physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole. Public Release Statement A: Approved for Public Release 27

28. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Military Definitions of SoS Academic System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. DoD System-of-systems: a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities. Both individual systems and SoS conform to the accepted definition of a system in that each consists of parts, relationships, and a whole that is greater than the sum of the parts; however, although an SoS is a system, not all systems are SoS. Public Release Statement A: Approved for Public Release 28

29. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 In Conclusion  Academic definitions of Systems, Systems-of- Systems, Measures, and Metrics have Military Domain Counterparts.  A working knowledge of the terms are critical to bridge the audiences when addressing different communities or domains.  Do not introduce new terminology, or redefine existing terms, if it is already authoritatively defined.  Introduce new terms in light of new concepts only if necessary. 29 Public Release Statement A: Approved for Public Release

30. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 In Conclusion Original “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” (Walker, 2005) Revised “By using only the MOP of shipping sunk, they were not able to ascertain the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have developed an MOE representative of the relationship between British public opinion, grain supply levels, and the availability of bread. Another useful MOE might have assessed American war intentions. These MOEs would have assessed the extent to which their capability achieved the desired end state.” Public Release Statement A: Approved for Public Release 30

31. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 Thank You! Thomas Holland, Ph.D. Naval Surface Warfare Center Dahlgren, VA 31 Public Release Statement A: Approved for Public Release

32. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 References  ASN(RDA) Chief Systems Engineer, 2006. Naval “Systems of Systems” Systems Engineering Guidebook, Vol. II.  Defense Acquisition University, 2013. Glossary of Defense Acquisition Acronyms and Terms. [Online] Available at: https://dap.dau.mil/glossary/Pages/GlossarySearch.aspx?rec=MOE  Fisher, D. A., 2006. An Emergent Perspective on Interoperation in Systems of Systems, Pittsburgh: Carnegie-Mellon Software Engineering Institute.  Maier, M. W., 1998. Architecting Principles for Systems-of-Systems. In: Systems Engineering. pp. 267-284.  Meyers, B. C., Monarch, I. A., Levine, L. & Smith, J. D., 2005. Including Interoperability in the Acquisition Process, Pittsburgh: Carnegie Mellon University, Software Engineering Institute.  Moffat, J., 2003. Complexity Theory and Network Centric Warfare, Washington, DC: DoD Command and Control Research Program. 32 Public Release Statement A: Approved for Public Release

33. O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5 References continued  Office of the Deputy Under Secretary of Defense for Acquisition and Technology, Systems and Software Engineering, 2008. Systems Engineering Guide for Systems of Systems. Version 1.0 ed. Washington, DC: ODUSD(A&T)SSE.  Office of the Deputy Under Secretary of Defense for Acquisition and Technology, 2008. Systems and Software Engineering. Office of the Deputy Under Secretary of Defense for Acquisition and Technology.  Oxford, 2013. Oxford Dictionaries. Available at: http://oxforddictionaries.com/  Shah, N. B., Rhodes, D. H. & Hastings, D. E., 2007. Systems of Systems and Emergent System Context. Hoboken, NJ  Smith, J. D., 2006. Topics in Interoperability: Structural Programmatics in a System of Systems, CMU/SEI-2006-TN-037, Pittsburgh: Carnegie Mellon Software Engineering Institute.  Walker, D. H., 2005. Developing Metrics in the Global War on Terrorism, Newport, RI: Naval War College. 33 Public Release Statement A: Approved for Public Release