Cumulative IOT&E Results Through FY 2008 Ideal Effective Number Effective And Suitable FY08 Suitable Total: 28% of Systems Not Suitable 2007: 4 of 8 (50%) Not Suitable 2008: 2 of 6 (33%) Not Suitable Cumulative Number of BLRIP Reports

DSB DT&E Taskforce Main Conclusion May 2008 “ . . . the single most important step necessary to correct high suitability failure rates is to ensure programs are formulated to execute a viable systems engineering strategy from the beginning, including a robust reliability, availability, and maintainability (RAM) program, as an integral part of design and development. No amount of testing will compensate for deficiencies in RAM program formulation.”

Section 231 Report to Congress Core T&E Principles T&E should concentrate on measuring improvements to mission capability and operational support based on user needs; T&E programs should experiment . . . . learn and understand the strengths and weaknesses of a system and its components, and the effect on operational capabilities and limitations; DT and OT activities should be integrated; T&E should begin early, be more operationally realistic, and continue through the entire system life-cycle; Evaluation should be conducted in the mission context expected at time of fielding to the user . . . in terms of operational significance; Evaluations should include a comparison against current mission capabilities; Evaluations should take into account all available data and information; T&E should exploit the benefits of appropriate M&S.

New Acquisition/T&E Policies McQueary/Finley Memo on Assessment of Op Test Readiness (21 May 2007) The DUSD(A&T) shall conduct an independent Assessment of Operational Test Readiness (AOTR) for all ACAT ID programs and special interest programs designated by the USD(AT&L) The CAE shall consider the results of the AOTR prior to making a determination of materiel system readiness for IOT&E.

New Acquisition/T&E Policies Young Memo on Competitive Prototyping (19 Sep 2007) All acquisition strategies requiring USD (AT&L) approval must be formulated to include competitive, technically mature prototyping through MS B.

New Acquisition/T&E Policies Young-McQueary T&E Policy Letter - (22 Dec 2007) DT and OT test activities shall be integrated and seamless Evaluations shall include a comparison with current mission capabilities T&E should assess improvements to mission capability and operational support based on user needs To more effectively integrate DT and OT, evaluations shall take into account all available and relevant data and information, including contractor data Operational evaluators will continue to fulfill their statutory roles in providing assessments of operational effectiveness, operational suitability, and survivability to the Milestone Decision Authority To realize the benefits of modeling and simulation, T&E will be conducted in a continuum of live, virtual, and constructive environments.

New Acquisition/T&E Policies McQueary-Finley Memo on Reliability Improvement WG (15 Feb 2008) Ensure programs are formulated to execute a viable systems engineering strategy, including a RAM growth program. Ensure government organizations reconstitute a cadre of experienced T&E and RAM personnel. Implement mandated integrated DT and OT, including the sharing and access to all appropriate contractor and government data and the use of operationally representative environments in early testing.

New Acquisition/T&E Policies McQueary-Finley Memo defining Integrated Testing (May 2008) “Integrated testing is the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, and evaluation.”

New Acquisition/T&E Policies Young Memo on RAM Policy (July 2008) The Service Secretaries are directed to establish Service policy to do the following: Effective collaboration between the requirements and acquisition communities Development contracts and acquisition plans must evaluate RAM during system design. Evaluate the maturation of RAM through each phase of the acquisition life cycle.

Senior Leadership Buy-In of New Reliability/T&E Policies “Having performance is important, but not as important in most cases, as having reliability.” - Hon. Donald Winters, Secretary of the Navy (Sept 3, 2008)

Initiatives to Improve Reliability, Maintainability, and Availability Reliability, Availability, Maintainability and Cost Manual (RAM-C Manual) DOT&E on JCIDS Functional Control Boards GEIA Standard 009, RFP and Contract Language, Investment Model Reliability Growth in design phase Ownership Cost Number of Failures in the Field RAM growth monitoring for incentives, Young/Bolton memos RAM program Evaluation and Standards, testing KPP RAM field data collection, feedback Right Contract and Incentives Right Design and Redesign Right Validation Right Requirements Right Development Right Next Increment

Basic Model R2 = .81 Improvement Ratio Investment/APUC = Major System Investment = Reliability Improvement Ratio 0.3659 2.119 X APUC 1 0.8 0.6 0.5 0.4 0.3 0.2 10 8 6 5 4 3 2 Improvement Ratio 7 100 80 60 50 40 30 20 1000 700 500 400 300 200 2000 Investment/ APUC R2 = .81 Improvement Ratio To create the basic model, we developed a cost estimating relationship (CER), shown in here by performing regression analysis on data from 17 programs. Required investment increases linearly with average production unit cost (APUC) and as a power of the reliability improvement ratio. APUC is essentially normalizing for program size and complexity. The reliability improvement ratio is (New MTBx − Old MTBx)/Old MTBx MTBx can be mean time between failure, mean time between removal, mean time between system abort, mean cycles between removal, or other similar measure relevant to a specific program. The R2 for this equation is 0.81, meaning 81% of the variability among the data can be accounted for. The required investment increases linearly with APUC and a power of the reliability improvement ratio. The CER appears to be valid across technologies, across different types of weapon systems, and across a wide range of complexity, from components to subsystems to complete platforms. Investment/APUC

Investment in Reliability Improvement, $M Notional Example Effect of Reliability Investment on System Cost (UAV) 500 1000 1500 2000 2500 3000 20 40 60 80 100 120 Investment in Reliability Improvement, $M 20-Year Cost, $M 20-year Cost, $ M example is based on a notional unmanned aerial vehicle. used the basic model to relate investment in reliability to achieved reliability, used the dynamic model to determine how many UAVs would have to be procured to ensure that 100 are available at a 95 percent confidence level translated that into total cost using the CASA model. (In this specific case, we regressed the support cost output from the CASA model against a range of reliability values and then used the result of the regression to estimate support cost as a function of reliability. calculated production cost directly from the number of UAVs. Total cost comprised the investment in reliability plus the cost to produce the required number of UAVs plus 20 years of support for each UAV Chart shows relatively modest investments in reliability relative to production unit cost produce large reductions in total cost. Beyond a 5:1 ratio of investment to production unit cost, the return on investment gradually decreases. Over the range of investments shown on the chart, there is always a return on investment. However, the curve will bend back up when investments in reliability do not return savings of at least 1:1. Investment in Reliability, $ M