1. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force 1 Changes in Cost Estimating John Cargill, GS-1515 AFCAA/FMAE ASMC RPDI March 8, 2012 Version 1.0.2

2. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Climate 2 Times they are a’changing There is a strong movement towards data driven estimates require databases increased use of CERs (cost estimating relationships) in addition to analogies and engineering estimates All three serve as independent crosschecks to each other Depending upon the center, one may be favored over the others Defense Cost Analysis and Resource Center (DCARC) has become the big data repository in the sky MilStd 881 Work Breakdown Structure Quality of data in the 1921’s and 1921-1’s have greatly improved

3. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Acknowledgements: Collaboration Thanks to Kristy Golden (AFCAA/FMAE) for her letting me take on the Missile Sufficiency Review Handbook and Database update Dr. Lew Fichter and Jeff McDowell, of Tecolote Research, for providing extensive input for potential improvements to the previous editions of the MSSRHbk and accommodated most of my “druthers.” John Odum (AAC/EBJ) and Bill Bolen (AAC/FMC) for their “wouldn’t it be nice if…” comments during the JASSM/JASSM-ER cost estimate. They will notice some of their wishes have been incorporated Ron Forch (AAC/FMC) for spending three months digging through Air Force technical manuals for specifications, weights and exploded diagrams 3

4. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Example Databases That Can Be Queried Unmanned Spacecraft Cost Model – now up to USCM9 NASA-Air Force Cost Model – NAFCOM Civil Engineers have many cost models and databases Realtors have updated models for various neighborhoods all around the country – independent variables: Neighborhood Bedrooms Sqft Age Number of cars in garage 4 Parametric query In a nutshell, we want something like this for costing missiles

5. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Fearlessly Going Where No Man Has Gone Before Outcomes Database & Viewer Multiple sources of conflicting data Structured query capability 5 Purifying, inflating & normalizing Hollering, yelling, swearing & going blind Bucketing, spreading & splitting Happy, happy, happy Fewer sleepless nights Relatively easy access to structured data

6. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Background: Building the Database History: CR-1147 July 2002 CR-1461 “Missile Systems Sufficiency Review Handbook” January 2010 CR-1461/1 “Missile Systems Sufficiency Review Handbook” October 2011 6 MSSRHbk is no longer a local product – it has become the standard with the U.S. Army and the U.S. Navy as well will us at AAC in the Air Force

7. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Acknowledging the Graphics 7 Where you see pretty pictures, I got them from my Tecolote cohorts in the MSSRHbk update project Where you see ugly pictures, those are mine

8. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Most Data Came from DCARC 8

9. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Nothing in Isolation! 9 All the Acronyms you’ll ever need: DCARC – Defense Cost & Resource Center (DoD) JCARD – Joint Cost Analysis & Research Database( NAVAIR) CRUAMM – Air Force Cost Risk and Uncertainty Metrics Manual (AFCAA) ACDB Automated Cost Database (AFCAA)

10. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Acquisition Phases Each Development phase data point is named using the current DoD 5000 series phase terminology of the respective period Efforts typically performed between Milestones B and C (SDD, EMD, FSED, FSD, Engineering Development, etc.) represent costs of an EMD/SDD phase Efforts typically performed between Milestones A and B (Dem Val, PDRR, Risk Reduction, Advanced Development, etc.) are deemed to represent the cost of a Technology Development phase Efforts typically performed beyond Milestone C (P3I and ECP) are deemed to represent the cost of a P3I phase Schedules referenced to CDR 10

11. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Data Normalization & Mapping: Meticulous Pain and Agony 11 Faux data

12. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Database The work breakdown structure was significantly expanded to increase the granularity of hardware subsystems New data incorporated Air vehicle weight Length Diameter (U.S. Army has found high correlation between diameter and cost) Fly-out Range Speed “Box-level” “SWAP” and cost Data to support analogies, CERs, and output based simulation Schedule database 12

13. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Database continued Continued alignment with typical Air Force estimating structures Completely mapable to MilStd 881 Meets the requirements for electronic transfer to the Navy’s Joint Cost Analysis Research and Database (JCARD) Viewer supports many sort, search, conversion and other functions Automatically creates an extensive series of statistical summaries Supports easy benchmarking by programs Creates schedule graphs referenced to CDR for programs selected Exports charts to MS Word 13

14. I n t e g r i t y - S e r v i c e - E x c e l l e n c e The Magic Tool 14 1.Contains most all the goodness known to man 2.Better than sliced bread

15. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Program Selection Programs to be included in the data set for analysis can be selected on the basis of Weight, length, diameter, range, speed Platform Air Surface Mission Air Surface Missile Defense 15 Allows asking parametric questions like: What would a 1000lb class subsonic air to ground cruise missile that can go 300 miles cost?

16. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Program Selection By Type of Guidance Guidance Multi-mode TERCOM INS Active Radar Semi-active Laser INS/GPS IR EO Passive Radar 16 Laser EO Passive Radar Laser IIR Pre-programmed Semi-active Radar Command

17. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Program Selection By Type of Warhead 17 Warhead Blast/Frag Nuclear HE Bomblets Non-nuclear kill vehicle Modular General Purpose Bombs Direct Frag Penetration HE Blast HETSC HE Blast/Frag Penetrator HE Tandom Shaped Charge Varies Penetrating Blast Frag Penetrator HESC Shaped Charge Explosively Formed Penetrator Multi-effects HEBF

18. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Program Selection By Propulsion, Lead Service, Control 18 Propulsion Engine/Solid Engine Solid None Control TVC Aero Surfaces None Gas Reaction Control TVC and Aero Surfaces Fluid-injection Jet Vanes Lead Service Navy Air Force Army Selecting: 1. No propulsion 2.Air Force yields 3.Aero surfaces GBU-15, JDAM, SDB I and SDB II

19. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Program Selection From Program List 19 Fair warning: You MUST see a program in the “Systems to Include” list BEFORE you hit “Save Picks…” or no program will be selected.

20. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Program Selection From Pre-determined Program Lists 20

21. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Of Course You Can Choose Any Base Year You Want 21 Caveat: Currently, only DoD Inflation is automatically supported for 3600 and 3020

22. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Example: Air-To-Surface Weapons 22

23. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection RDTE Burn Rate 23 A good bench-marking chart Compare program of interest against other programs in the database Compare program of interest against programs selected If your program is an outlier on burn rate, maybe you have ramped up too slowly, two quickly or have an extraordinary schedule Compare by RDT&E Phase Synthesized Data

24. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: RDTE Nonrecurring Factor of Recurring Cost 24 Synthesized Data

25. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: RDTE Factor of System PMP Cost 25 Toggled for RDTE Factor of System PMP Cost Can toggle for RDTE Factor of Air Vehicle PMP Cost Synthesized Data

26. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Ground Selection: Production Non Recurring Factor of Recurring Cost 26 Synthesized Data

27. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Ground Selection: Production Non Recurring Factor of Recurring Cost 27 Toggled for Production Factor of Air Vehicle PMP Cost Can toggle for Production Factor of System PMP Cost Synthesized Data

28. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Production Unit cost (T1-UC1000) 28 Since T1 is often greatly affected by minor changes in CIC, options are provided for Unit Costs at 100, 500, 1000 or User Entered

29. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Production Unit cost (T1) 29 Toggled for Production Unit Cost (T1) Can toggle for Production Unit Cost Several Other Ways Synthesized Data

30. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Production Unit Cost per Pound (T1-UC1000) 30 Toggled for Production Unit Cost (T1) Can toggle for Production Unit Cost Several Other Ways Synthesized Data

31. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: RDTE Nonrecurring Cost per Pound (T1-UC1000) 31 Toggled for RDTE Nonrecurring Cost per Pound (T1) Can toggle for RDTE Nonrecurring Cost per Pound Several Other Ways Synthesized Data

32. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: RDTE Nonrecurring Cost per T1-T1000 32 Toggled for RDTE Nonrecurring Cost per T1 Can toggle for RDTE Nonrecurring Cost Several Other Ways Synthesized Data

33. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Crosschecks 33 Toggled for Default CIC Options and Default RDTE Phase Options Synthesized Data

34. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Crosschecks continued 34 Toggled for Default CIC Options and Default RDTE Phase Options Synthesized Data

35. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Crosschecks continued 35 Toggled for Default CIC Options and Default RDTE Phase Options Synthesized Data

36. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Production Phase Cost Improvement Curves 36 Cost Improvement Curves “Learning Curve” -- database uses unit theory “Rate Surface” -- combination of learning and rate which is very important with lot buys of varying sizes Synthesized Data Note for Regression Experts: Coefficient of Variation is greater for rate slope than learning – implies that economic order quantities are not being followed historically, or possibly there is multicollinearity between the rate and cumulative quantity.

37. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Air-To-Surface Selection: Production Phase Proportions 37 Synthesized Data Yes, Virginia! There are Non-Recurring costs during production, especially for the first lot

38. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule Analysis 38 This iteration of the MSSRHbk with its Database Viewer is First version to reference program events to how much before or how much after CDR CDR has the highest correlation with other events Milestones and PDR move around a lot New DoD guidance is that PDR will occur before MS B, whereas in the past it usually occurred after MS B Program picking (selection) is separate from the programs picked for costs and the other metrics because it may be necessary to eliminate some programs for cost variances resulting from funding vagaries, etc., yet the schedule might be appropriate to retain It will not be explained why this button is called “Bonus Schedule Analysis”

39. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule Analysis Air-To-Surface 39 Same Air-To-Ground program picks as in the cost examples but with a slight twist: You get to select whether you want to work with Median, Mean or Lognormal Mean schedule durations

40. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule 40 Synthesized Data PDR MS A MS B

41. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule: 41 Synthesized Data CDRCDR CDR is a real point in time independent of acquisition and milestone strategy – it is a time when most of the engineering drawings are complete and we are ready to start “bending tin”

42. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule: CDR Reference 42 Synthesized Data

43. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule Analysis: MS B Referenced 43 For planning, comparison and cost estimating purposes it is still important to be able to estimate times from MS B, even though the requirements to enter MS B have changed Be careful how you use the MS B reference as it is more applicable for reviewing historic programs where PDR and CDR occurred after MS B For modern programs where risk reduction efforts result in System-level PDR and Component-level CDR, you should use the CDR-referenced times MS B referenced time lines can serve as a rough estimate of the ceiling, as one would expect modern programs to take less time post MS B because more is accomplished before MS B It will not be explained why this button is called “Bonus Schedule Analysis”

44. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Schedule: MS B Reference 44 Synthesized Data

45. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Exporting Summary Statistics 45 One of the new features of the viewer is the ability to export the entire set of summary statistics charts to MS Word, which may be subsequently pasted to MS Power Point briefing charts Cell B2 of the Analysis tab has a button that causes exporting the summary charts to MS Word One feature of this export is that tables are produced for the entire database and for the programs selected

46. I n t e g r i t y - S e r v i c e - E x c e l l e n c e One Last Feature: Pages of Systems Descriptions 46

47. I n t e g r i t y - S e r v i c e - E x c e l l e n c e One More Last Feature: Lots of Exploded Drawings 47

48. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Conclusions 48 User friendly database and viewer More time analyzing and thinking Less time grunting With viewer, it is convenient to make comparisons and talk to Program Office Ideally, database and viewer would be Oracle or Access based, however, Excel works reasonably well in the interim

49. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Take Home Message 49 Early adoption of this database technology has brought us closer to our information sources and is allowing us to work smarter, not harder Early Adoption of Technology Closer to data sources Working smarter, not harder

50. I n t e g r i t y - S e r v i c e - E x c e l l e n c e Questions and Point of Contact John Cargill GS-1515 Air Force Cost Analysis Agency Operating Location Eglin AFB, FL 850-883-3460(w) 703-371-0891(c) john.cargill@eglin.af.mil 50

51. I n t e g r i t y - S e r v i c e - E x c e l l e n c e 51 The End

52. I n t e g r i t y - S e r v i c e - E x c e l l e n c e 52 BACKUP SLIDES