1. © 2011 Carnegie Mellon University QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation Presenters:Dave Zubrow PhD Bob Ferguson (SEMA) Date:November 3, 2011 Location:COCOMO Forum 2011

2. 2 © 2011 Carnegie Mellon University DOD Acquisition Process and GAO Knowledge- Based Acquisition Practices Source: GAO 11-499T “Establishing realistic cost and schedule estimates that are matched to available resources: Cost and schedule estimates are often based on overly optimistic assumptions. Our previous work shows that without the ability to generate reliable cost estimates, programs are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. Inaccurate estimates do not provide the necessary foundation for sufficient funding commitments. Engineering knowledge is required to achieve more accurate, reliable cost estimates at the outset of a program.

3. 3 © 2011 Carnegie Mellon University DOD Acquisition Process and GAO Knowledge- Based Acquisition Practices Source: GAO 11-499T

4. 4 © 2011 Carnegie Mellon University ABC Approval Acquisition Phases and Decision Milestones Technology Development Engineering & Manufacturing Production & Deployment N Cost Estimate Based on: Limited Information Expert Judgment Analogies Est. $$$Est. $$Est. $ Delay Y Materiel Solution Challenges: 1) change and uncertainty 2) optimistic judgment Estimating the Cost of Development at Milestone A

5. 5 © 2011 Carnegie Mellon University QUELCE for Producing Milestone A Estimates Brainstorm Change Drivers and Define States Develop Cause and Effect Matrix of Change Drivers Rate Relationships, Restructure and Reduce using DSM Produce BBN Model of Reduced Matrix Assign Probabilities and Conditional Probabilities to Nodes in BBN Define Scenarios of Program Execution Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions Map BBN Change Factor Output States to COCOMO Cost Driver Values

6. 6 © 2011 Carnegie Mellon University Change Drivers and States Factors seeded by, but not limited to, Probability of Program Success (PoPS) factors.

7. 7 © 2011 Carnegie Mellon University Cause and Effect Matrix for Change Drivers Each cell gets a value (blank, 1, 2, or 3) to reflect the perceived cause-effect relationship of the row heading to the column heading) Note: The sum of a column represents a dependency score for the column header. The sum of a row is the value of the driving force of the row header

8. 8 © 2011 Carnegie Mellon University Reduced Cause and Effect Matrix Use Design Structure Matrix techniques to reduce so an acyclic graph can be produced. Indicates remaining cycle that must be removed

9. 9 © 2011 Carnegie Mellon University BBN of Reduced Cause and Effect Matrix Translate the C-E Matrix into a BBN. Orange nodes are program change factors. Green nodes are outputs that will link to COCOMO cost drivers. These output nodes were selected as an example and represent sets of COCOMO cost drivers.

10. 10 © 2011 Carnegie Mellon University Assign Probabilities and Conditional Probabilities to BBN Nodes Use expert judgment to assign probabilities and conditional probabilities to the nodes. These assignments could also be empirically based if the data are available. Capability Definition is affected by CONOPS and Strategic Vision

11. 11 © 2011 Carnegie Mellon University Define Scenarios of Program Execution Scenarios for alternate futures specify nominal or non-nominal states for selected change drivers to test alternative results.

12. 12 © 2011 Carnegie Mellon University Map BBN Change Factor Output States to COCOMO Cost Driver Values BBN output states are mapped to values of COCOMO cost drivers. Currently done with expert judgment. Later could be done using a data-based algorithm. Distributions of BBN outputs used in next step.

13. 13 © 2011 Carnegie Mellon University Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions Using distribution of BBN outputs, Monte Carlo simulation is used to produce the distribution of the cost estimation for each defined scenario. Mapped COCOMO value 4 4 BBN Outputs

14. 14 © 2011 Carnegie Mellon University Task 2: Develop Efficient Techniques To Calibrate Expert Judgment of Program Uncertainties Solution Calibrated Un-Calibrated Estimate of SW Size Domain-Specific “reference points” 1)Size of ground combat vehicle targeting feature xyz in 2002 consisted of 25 KSLOC Ada 2)Size of Army artillery firing capability feature abc in 2007 consisted of 18 KSLOC C++ 3)… Step 1: Expert virtual training of “reference points” Step 2: Expert takes series of domain specific tests Step 3: Expert reduces overconfidence Step 4: Expert renders calibrated estimate of size Calibrated = more realistic size and wider range to reflect true expert uncertainty

15. 15 © 2011 Carnegie Mellon University Next Steps Classroom experiments with software & systems engineering graduate students Structured feedback for early validation & refinement of our method, initially: – Selected steps at least through reducing the client’s dependency matrix at the University of Arizona – Calibrating expert judgment & reconciliation of differences in judgment at Carnegie Mellon University Invitation to Participate Many pieces and parts to test and improve Empirically validating the overall approach – Retrospective: projects that were estimated and have recorded change history Testing BBN output parameters and mapping to estimation parameters Fitting to additional estimation tools and cost estimation relationships (CER) Do new estimates properly inform decisions about risk and change management? (secondary benefit)

16. 16 © 2011 Carnegie Mellon University Contact Information Presenters/ Points of Contact Dave Zubrow SEMA Telephone: +1 412-268-5243 Email: dz@sei.cmu.edudz@sei.cmu.edu Bob Ferguson SEMA Telephone: +1 412-268-9750 Email: rwf@sei.cmu.edurwf@sei.cmu.edu Web: www.sei.cmu.edu www.sei.cmu.edu/measurement/ U.S. mail: Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257

17. 17 © 2011 Carnegie Mellon University NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks in this presentation is not intended in any way to infringe on the rights of the trademark holder. This Presentation may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.permission@sei.cmu.edu This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013.