1. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology System of Systems Engineering Cost Modeling: What Makes It Different from Traditional Systems Engineering Cost Modeling Jo Ann Lane USC CSSE jolane@usc.edu Ricardo Valerdi MIT rvalerdi@mit.edu COCOMO Forum October 2007

2. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE2 Overview Overview of SoSs and SoSE Summary of key comparative research and pilot studies Comparison of traditional SE and SoSE cost models Conclusions

3. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE3 Relationships between Traditional Systems, SoSs, and Complex Systems [Kuras and White, INCOSE, 2005]

4. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE4 What is a “System of Systems”? Very large systems developed by creating a framework or architecture to integrate component systems SoS component systems independently developed and managed –New or existing systems in various stages of development/evolution –Have their own purpose –Can dynamically come and go from SoS SoS exhibits emergent behavior not otherwise achievable by component systems Typical domains –Business: Enterprise-wide and cross-enterprise integration to support core business enterprise operations across functional and geographical areas –Military: Dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment Based on Mark Maier’s SoS definition [Maier, 1998] For more on SoS definitions, see [Lane and Valerdi, 2007]

5. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE5 TSE, SOSE, and Related Industry Standards Current standards for TSE processes –ISO/IEC Standard 15288 – describes system processes from system conception through system retirement –ANSI/EIA Standard 632 – detailed processes for conceptualization, development and transition to operation (subset of ISO/IEC 15288) –SEI’s Capability Maturity Model Integrated – framework for defining an organization’s standard processes and procedures –Defense Acquisition Guidebook (DAG) – references above as examples of best practices, but defines own set of processes Guidelines for SoSE –SoS SE Draft Guidebook (extension to DAG)

6. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE6 SoSE Compared to TSE Activities SoSs have their own characteristics and associated challenges that are different from traditional systems and large, complex systems Reported areas of difference –Architecting –Prototypes/experimentation/tradeoffs –Scope of SoS –SoS performance –Maintenance and evolution Key challenges for DoD SoSE –Business model and incentives to encourage working together (SoS level) –Doing the necessary tradeoffs at the SoS level –Human-system integration –Commonality of data, architecture, and business strategies (SoS level) –Removing multiple decision making layers –Requiring accountability at the enterprise level –Evolution management –Maturity of technology

7. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE7 Recent Research Findings* Many types of SoS SoS Engineering Teams: Varying degrees of responsibility and authority Incorporating many agile-like approaches to handle –Multiple constituent systems –Asynchronous activities and events –Quickly take advantage of opportunities as they appear SoSE must –Support multiple purposes and visions –Requires significantly more negotiation –Is content to satisfice rather than optimize SoSE activities map to traditional SE activities (e.g., DAG and EIA 632), but take on a different focus and scope * Based on USC CSSE SoSE cost model research, MIT Systems Engineering Advancement Research Initiative, and OSD SoS SE pilot studies

8. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE8 Relationship Among Core Elements of SoS SE Translating capability objectives Developing, evolving and maintaining SoS design/arch Understanding systems & relationships (includes plans) Assessing (actual) performance to capability objectives Addressing new requirements & options Monitoring & assessing changes Orchestrating upgrades to SoS External Influences Typically not the role of the SE but key to SoS [assume these are fixed] Block upgrade process for SoS Persistent framework overlay on systems in SoS [architecture] Large role of external influences

9. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE9 SoSE Core Element Mapping to COSOSIMO Sub-models Translating capability objectives Developing, evolving and maintaining SoS design/arch Understanding systems & relationships (includes plans) Assessing (actual) performance to capability objectives Addressing new requirements & options Monitoring & assessing changes Orchestrating upgrades to SoS Planning, Requirements Management, and Architecting (PRA) Source Selection and Supplier Oversight (SO) SoS Integration and Testing (I&T) COSOSIMO

10. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE10 Comparison of System of Interest Focus Areas AreaTraditional SystemsSystem of Systems What to engineer Based on a set of functional and performance requirements for the system of interest Based on a set of SoS capabilities that are then translated into high level requirements for further analysis A single capability can result in multiple requirements that affect multiple constituent systems View of system-of- interest Clear system boundaries Interfaces Systems that contribute to SoS capabilities and the interrelationships between those systems ArchitectureDeveloped and optimized to support single purpose of system Net-centric, focused on information sharing Does not address design details within constituent systems, but rather the way the systems work together to meet user needs Sufficient versus optimized Design approach Often top-downCombined top-down and bottom-up, with focus on –Existing assets (systems) that are within the SoS –Opportunities within constituent system lifecycles for changes

11. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE11 Comparison of System of Interest Focus Areas (continued) AreaTraditional SystemsSystem of Systems ImplementationContract-controlled, often using an incremental, evolutionary, or spiral process Focus on total system SoS functionality implementation accomplished through combination of negotiation, sometimes funded by SoS or system owner, not always done via formal agreements Asynchronous and incremental due to lifecycles of constituent systems Primarily concerned with the implementation of SoS functionality, Monitors the evolution of constituent systems to ensure that SoS is not adversely impacted, but not typically involved in the implementation details TestingTraditional testing activities, e.g., DT&E and OT&E Attempt to leverage off of constituent system testing Often impossible to test full-up SoS in a lab—often rely on constituent system integration labs and operational testing Operationally, looking for how users use the system and identifying emergent behavior for further analysis

12. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE12 Areas of Emphasis in SE and SoS SE * AreaTSESoS SE PurposeDevelopment of a single system to meet stakeholder requirements and defined performance Evolving new systems of systems capability by leveraging synergies of legacy systems and emerging capabilities Systems Architecture Established early in the life cycle; expectation set remains relatively stable Dynamic adaptation as emergent needs change System Interoperability Interface requirements are defined and implemented for the integration of components in the system Component systems can operate independently of SoS in a useful manner; protocols and standards are essential to enable interoperable systems System “ilities”Reliability, maintainability, and availability are typical concerns Enhanced emphasis on ilities such as flexibility, adaptability, and composability Acquisition and Management Centralized acquisition and management of the system Component systems separately acquired and continue to be managed and operated as independent systems Anticipation of Needs Concept phase activity to determine system needs Intense concept phase analysis followed by continuous anticipation, aided by ongoing experimentation FundingSingle or homogenous stakeholder group with stable cost/funding profile and similar measures of success Multiple heterogeneous stakeholder groups with unstable cost/funding profile and measures of success * [Valerdi, et al., 2007]

13. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE13 Engineering Cost and Schedule—What Do Engineering Cost Models Look At? Engineering product characteristics Processes used to develop the product Skills and experience levels of the technical staff responsible for development of product Size drivers used to compute nominal effort Cost drivers used to adjust nominal effort up or down

14. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE14 Comparison of Cost Model Parameters Parameter AspectsCOSYSMOCOSOSIMO Size drivers# of system requirements # of system interfaces # operational scenarios # algorithms # of SoS requirements # of SoS interface protocols # of constituent systems # of constituent system organizations # operational scenarios “Product” characteristicsSize/complexity Requirements understanding Architecture understanding Level of service requirements # of recursive levels in design Migration complexity Technology risk #/ diversity of platforms/installations Level of documentation Size/complexity Requirements understanding Architecture understanding Level of service requirements Component system maturity and stability Component system readiness Process characteristicsProcess capability Multi-site coordination Tool support Maturity of processes Tool support Cost/schedule compatibility SoS risk resolution People characteristicsStakeholder team cohesion Personnel/team capability Personnel experience/continuity Stakeholder team cohesion SoS team capability

15. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE15 Impact of Differences to SE Cost Model Elements of SE that are not of major concern/driver of effort or can be handled as a constant –Number of system algorithms –Number of recursion levels in the design –Migration complexity –Number/diversity of platforms/installations –Level of documentation –Multi-site coordination Elements of SoS SE not adequately addressed in SE cost model –Analysis of capabilities to determine SoS and constituent system requirements –Impact of negotiations required to develop architecture and enhancements –Impact of number of constituent systems and organizations that own and manage the constituent systems –The asynchronous nature of constituent component changes and the accommodation of partial increment implementations operationally –Additional effort required to accommodate unplanned constituent component changes –Uncertainty with respect to funding sources and impact on required negotiations

16. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE16 Conclusions to Date and Future Work Recent research efforts articulate some significant differences between TSE and SoSE –At a basic level, many of the activities are the same –However, there are significant differences to the scope and focus of the engineering activities Major focus of/impacts to SoSs –Legacy systems and the independent control of them –Focus on inter-relationships between relatively independent components vs. integration of tightly coupled components Elements of SoS SE not adequately addressed in existing traditional SE cost models –COSOSIMO strives to address these SoS SE differences –We need industry support to better understand and calibrate these differences

17. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology Backup Charts

18. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE18 SoSE Compared to Traditional SE Activities: Reported Differences Architecting –Architecting composability vs. decomposition (Meilich 2006) –Net-friendly vs. hierarchical (Meilich 2006) Prototypes/experimentation/tradeoffs –Early tradeoffs/evaluations of alternatives (Finley 2006) –Intense concept phase analysis followed by continuous anticipation; aided by ongoing experimentation (USAF 2005) –Modeling and simulation, in particular to better understand “emergent behaviors” (Finley 2006) –First order tradeoffs above the component systems level (e.g., optimization at the SoS level, instead of at the component system level) (Garber 2006) –Discovery and application of convergence protocols (USAF 2005)

19. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE19 SoSE Compared to Traditional SE Activities: Reported Differences (continued) Scope and performance –Added “ilities” such as flexibility, adaptability, composability (USAF 2005) –Human as part of the SoS (Siel 2006, Meilich 2006, USAF 2005) –Organizational scope defined at runtime instead of at system development time (Meilich 2006) –Dynamic reconfiguration of architecture as needs change (Meilich 2006) Maintenance and evolution –Component systems separately acquired and continue to be managed as independent systems (USAF 2005)

20. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE20 SoSE Core Element Descriptions Translating capability objectives –Developing a basic understanding of the expectations of the SoS and the core requirements for meeting these expectations, independent of the systems that comprise the SoS Understanding systems and relationships –In a SoS, the focus is on the systems which contribute to SoS SE capabilities and their interrelationships (as opposed to in a system, the focus is on boundaries and interfaces) Assessing actual performance to capability objectives –Establishing SoS metrics and methods for assessing performance and conducting evaluations of actual performance using metrics and methods Developing, evolving, and maintaining an SoS architecture/design –Establishing and maintaining a persistent framework for addressing the evolution of the SoS to meet user needs, including possible changes in systems functionality, performance or interfaces

21. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE21 SoSE Core Element Descriptions (continued) Monitoring and assessing changes –Monitoring proposed or potential changes and assessing their impacts to: Identify opportunities for enhanced functionality & performance, and Preclude or mitigate problems for the SoS and constituent systems (this may include negotiating with the constituent system over how the change is made, in order to preclude SoS impacts Addressing new requirements and options –Reviewing, prioritizing, and determining which SoS requirements to implement next Orchestrating upgrades to SoS –Planning, facilitating, integrating, testing changes in systems to meet SoS needs

22. University of Southern California Center for Systems and Software Engineering Massachusetts Institute of Technology October 2007©USC-CSSE22 References Dahmann, J. (2007); “ Systems of Systems Challenges for Systems Engineering”, Systems and Software Technology Conference, June 2007. DiMario, Mike (2006); “System of Systems Characteristics and Interoperability in Joint Command Control”, Proceedings of the 2nd Annual System of Systems Engineering Conference Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Finley, James (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Garber, Vitalij (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03 Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Kuras, M. L., White, B. E., Engineering Enterprises Using Complex-System Engineering, INCOSE Symposium 2005. Lane, J., Valerdi, R., “Synthesizing System-of-Systems Concepts for Use in Cost Modeling,” Systems Engineering, Vol. 10, No. 4, December 2007. Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284) Meilich, Abe (2006); “System of Systems Engineering (SoSE) and Architecture Challenges in a Net Centric Environment”, Proceedings of the 2nd Annual System of Systems Engineering Conference Pair, Major General Carlos (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference Proceedings of AFOSR SoSE Workshop, Sponsored by Purdue University, 17-18 May 2006 Proceedings of Society for Design and Process Science 9 th World Conference on Integrated Design and Process Technology, San Diego, CA, 25-30 June 2006 Siel, Carl (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California Valerdi, R., Ross, A., Rhodes, D., “A Framework for Evolving System of Systems Engineering,” CrossTalk - The Journal of Defense Software Engineering, October 2007. United States Air Force Scientific Advisory Board (2005); Report on System-of-Systems Engineering for Air Force Capability Development; Public Release SAB-TR-05-04