1. COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Making Connections to Existing CERs with QUELCE* COCOMO Forum November 2015 Bob Ferguson *Quantifying Uncertainty in Early LifeCycle Estimation

2. 2 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Copyright 2015 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721- 05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. [Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. Carnegie Mellon® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. DM-0003047

3. 3 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Session Agenda Overview of QUELCE CERs 1 in Estimation Connecting the QUELCE BBN 2 to CERs – Glue Nodes Summary 1 Cost Estimating Relationships 2 Bayesian Belief Network

4. 4 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited QUELCE QUELCE is a method for analyzing the effects of changing assumptions and other risks inherent in product development. Rather than focusing on the consequences of each risk, the method tries to analyze the cascading effects of multiple changes. The current presentation describes ways to connect the QUELCE probability network (BBN) to many different cost estimation relationships.

5. 5 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited QUELCE Intended to Inform CER Inputs (Example) Acquisition Environment A.1 Acquisition Category (ACAT) Status A.2 Governance, Policies, and Standards A.3 External Interdependencies / Coordination A.4 External Stakeholders A.5 External Events, Funding A.6 Other: Acquisition Environment A.7 Capability Definition Acquisition Management B.1 Acquisition Strategy B.2 Contracting B.3 Management Structure B.4 Program Scope, Requirements B.5 Budget B.6 Schedule B.7 Staffing B.8 Facilities, Support Technology, and Equipment B.9 Program Information Management B.10 Program-Contractor Performance B.11 Other: Acquisition Management Technical Solution C.1 Conceptual Design / Specification C.2 System Architecture and Design C.3 Production and Construction C.4 Certification and Accreditation C.5 Deployment, Operations, and Support C.6 Technology Maturity / Readiness C.7 Estimated Complexity / Difficulty C.8 Supply Chain Products C.9 Other: Engineering Solution / Work Products Need to map QUELCE change drivers (from the left) to the CER input parameters (on the right)

6. 6 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Modeling Uncertainty Complexity Reduction 1. Use QUELCE Repository to Populate Driver State Matrix 3. Develop BBN Model and Assign Conditional Probabilities to BBN Model 4. Calculate Cost Factor Distributions for Program Execution Scenarios 5. Monte Carlo Simulation to Compute Cost Distribution 2. Evaluate Cause and Effect Relationships and Reduce Explosion via Dependency Structure Matrix Overview of QUELCE Method Legend: QUELCE Change Driver Repository 1. Driver State Matrix 2. Dependency Structure Matrix 3. BBN Model 4. Cost Factor Distributions by Scenario of Change 5. Monte Carlo with Cost Estimation Tools SRDR DAES SARS

7. 7 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited CERs in Estimation CAPE recommends four cost estimation methods: Analogy: (early in program lifecycle -> MS A) Parametric (Statistical) - CERs : (early in program lifecycle -> MS B) Engineering (“bottoms up”): (MS C and later) Actual Costs: (LRIP and later) Different Types of Cost Estimation Relationships (CERs) Elements of scope – things we build Required process elements associated to scope Required purchases Scope Estimate of volume, weight, size, capacity based on required deliverable characteristics. This may be LOC, Function Points, requirements, etc… Product characteristics (quality attributes) such as speed of performance Quantity of deliverable Required Process Elements Associated to Scope Certifications

8. 8 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Methods Employed in CERs Scope Estimates CER is a parametric model - an equation used to estimate a given cost element using an established relationship with one or more independent variables. Analogy to past projects can be used to bound regression parameters. Expert judgment recommendations based on experience and data. Required Processes Associated to Scope Built from the scope estimates by using internal history and a proportional formula. Some of the required processes will be covered by a parametric tool. Other Required Processes Typically based on assigning a resource cost for duration of schedule or duration of development for a specific element of scope. Purchases Fixed cost based on count of need.

9. 9 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited What Are Glue Nodes? Definition A glue node is a way to connect the output from the BBN in step 3 of the QUELCE method to the CERs for your estimation model. Rationale Glue nodes are used to incorporate the front end uncertainty and scenario analyses afforded by QUELCE in a practical manner by mapping the output of the BBN into existing parameters in the CER. Example The process elements of scope are uncertain based on need for innovation, product complexity, etc. Hence, we attempt to identify the related nodes in the model. Those nodes are connected to a glue node. The probability distribution is calculated as the joint probability distribution of the parent nodes. The glue nodes are mapped to selected parameters of the CER.

10. 10 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited BBN Model with three Glue Nodes Product Challenge Program Challenge Scope (Size)

11. 11 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited How to Use Glue Nodes Each element of scope is likely to require different glue nodes Schedule The overall schedule will drive significant costs not associated to a specific scope, so different CERs are employed. Schedule will be specific to a lifecycle. Use The BBN must be connected to the selected CERs via glue nodes. The glue node embodies the joint probability of several other BBN nodes. The glue nodes will be specific to the CER. The BBN itself usually remains the same across different CERs. BBN Analysis Bayesia and AgenaRisk are example tools that support this function. It is easy to add the needed joint probability distributions.

12. 12 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited COCOMO II Example We used 3 glue nodes Size – Mike Ross code growth provides a useful size range – Median and a high percentile value are calculated Product Challenge – We would like to think of product domain as providing a critical combination such as real-time-embedded very high reliability – Also novelty or invention adds to complexity Program Challenge – Factors associated to contractor skill, program office flexibility, and number of contractors

13. 13 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited COCOMO II Example - Glue Nodes COCOMO parameter mapping -- Select parameters for “Product Challenge” and “Program Challenge.” Range is limited for each of the selected parameters. Map BBN glue node values to COCOMO parameter values. Some parameter scales had to be reverse ordered so that the effect had the appropriate direction Mapped COCOMO value BBN Outputs

14. 14 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Product Challenge Factors COCOMO Parameter XLVLLNHVHEH Scale FactorsPREC 12345 6.24.963.722.481.240 FLEX 543 5.074.053.04 RESL 2.53 TEAM PMAT Effort MultipliersPERS 54321 1.621.261.000.830.630.49 RCPX 12345 0.490.60.8311.331.912.72 PDIF 11345 0.8711.291.812.61 PREX FCIL RUSE 12345 0.9511.071.151.24 SCED Italicized values are derived from the BBN; corresponding COCOMO values are directly below the BBN values.

15. 15 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Program Challenge Factors COCOMO Parameter XLVLLNHVHEH Scale FactorsPREC FLEX RESL Nominal 2.53 TEAM Joint Norm 3.95 0.99 PMAT12345 reversedvalues 00.911.822.733.64 Effort MultipliersPERS fixedrange not modeled 0.83 1.62 RCPX PDIF PREX fixed ranges 1.12 0.74 FCIL 1.1 0.87 RUSE SCED reversedvalues 1-235 11.141.43 Italicized values are derived from the BBN; corresponding COCOMO values are directly below the BBN values.

16. 16 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Monte Carlo Modeling Mapped COCOMO value BBN Outputs Each CER parameter is assigned a distribution based on the Glue node distribution Monte Carlo simulation then enables a distribution of cost using the CER with input distributions

17. 17 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Summary Change Driver Scenarios modeled to inform CER inputs leading to cost distributions

18. 18 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Contact Information Robert Ferguson Senior Member of Technical Staff Software Solutions Division, SEAP Telephone: +1 412-268-1121 Email: rwf@sei.cmu.edu U.S. Mail Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA Web www.sei.cmu.edu www.sei.cmu.edu/contact.cfm Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257

19. 19 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited BACKUP SLIDES

20. 20 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Change Drivers Definition: A change driver is any assumption or decision that has the potential to change. Decisions: Selection of suppliers and contractors including test facilities Selection of technology or component Assumptions: A technology, currently at TRL 6, will mature sufficiently by Milestone C. A KPP goal (key performance parameter) can be achieved (KPP goals for software defined radios were not achieved). Scope and feature definitions are fixed by Milestone B.

21. 21 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Change Drivers and Dependency Structure Matrix “Change Drivers” The specific assumptions and risks identified by program subject matter experts (SME) SMEs are asked to make estimates of frequency of change The Dependency Structure Matrix (DSM) Relates changes by asking whether a second change event would be a consequence of an earlier one. The matrix tends to be very large and must be reduced and reorganized into a triangular matrix in order to create the probability network. SEI has an automated tool that makes this process much easier. It now takes about 30 minutes or less.

22. 22 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Modeling Schedule Impacts QUELCE Addresses Schedule Slip Use lifecycle synchronization points such as PDR and CDR. Delays are common and usually have identifiable causes that have been modeled in the BBN. The critical concern for schedule is whether specific change events occur that may cause schedule delays. Synchronization Concerns A critical dependency during the lifecycle is the need to synchronize development teams and tasks in order to do any form of product validation. Without this validation exercise, no forecast of delivery is trustworthy. Schedule forecasts of deliverables are essential to the accuracy of the cost estimate.

23. 23 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Schedule Example: Causes of Schedule Delay Supply Chain Obsolete parts Change of supplier Technology Readiness and Manufacturing Readiness e.g. GN (gallium nitride) is a potential replacement for Gas (gallium arsenide). Both electrical characteristics and manufacturing problems still need study. Software will interact with electrical characteristics. Contractor – Program disputes Approval of engineering change request CDRL review delays Limited availability of specific resources – Test equipment and facility Funding Delays … several others that are familiar to experienced program managers

24. 24 COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Bayesian Belief Network Directed Graph A directed graph is constructed from the triangular matrix. Each node gets a probability of occurrence and a probability of change effect from each parent node. Probabilities are generated by SMEs and by data from past programs. Stability of the BBN Our expectation is the BBN should change rarely during most programs. New risks or assumptions not tested would add a node. As time passes, more and more nodes would be set to zero expectation of change. BBN probabilities would consistently narrow over the course of time.