We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byAlissa Hails
Modified over 2 years ago
© 2009 The MITRE Corporation. All rights reserved. Integrating Developmental and Operational Testing Dr. William (Dave) Bell 2 December 2009
© 2009 The MITRE Corporation. All rights reserved. Agenda/Overview ■What/Why Integrated Testing? ■When Should We Do It? ■What Can/Should Testers Do? ■Summary 2
© 2009 The MITRE Corporation. All rights reserved. Recent DoD Integrated Testing Policy 3 Integrated testing is the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, evaluation and reporting by all stakeholders particularly the development (both contractor and government) and operational test and evaluation communities T&E expertise must be brought to bear at the beginning of the system life cycle to provide earlier learning about the strengths and weaknesses of the system under development
© 2009 The MITRE Corporation. All rights reserved. Integrated Testing Defined 4 ■Defense Acquisition Guidebook, Chapter 9 –Whenever feasible, DT&E and OT&E events should be combined, if that supports technical and operational test objectives to gain the optimum amount of testing benefit for reasonable cost and time. The user community should be involved early in test planning to ensure the statement of desired capabilities is interpreted correctly and tested realistically. Certain events can be organized to provide information useful to developmental and operational evaluators and lend themselves to the DT and OT approach. The concept is to conduct a single combined test program that produces credible, qualitative and quantitative information that can be used to address developmental and operational issues. Examples of this approach include: combined DT/OT events or piggybacking an operational assessment onto a developmental test. Likewise, developmental testing data requirements can be accommodated by an operational test. This approach can reduce the time and expense of conducting dedicated OT events that replicate DT events, or vice versa, yet still provide adequate technical risk reduction. The developmental and operational testers can develop a test management structure to share control of the combined events. Combined DT and OT events and test data requirements must be identified early to prevent unnecessary duplication and to control costs. It is important that neither the DT&E nor OT&E objectives are compromised in designing combined events. Combined DT & OT events support technical and operational objectives User community involved early to ensure desired capabilities interpreted correctly and tested realistically
© 2009 The MITRE Corporation. All rights reserved. DT/OT to IDT/OT Evolution - Past 5 Contractors Responsible Test Organizations Find & fix technical problems Operational Test Agencies Director, OT&E Discover ops issues & confirmation Developmental Test Operational Test Certification
© 2009 The MITRE Corporation. All rights reserved. Integrated Test DT/OT to IDT/OT Evolution - Present 6 Developmental Test Operational Test Certification OTAs DOT&E Finish discovery & perform confirmation Ktrs RTOs Find & fix technical problems Ktrs RTOs OTAs DOT&E Certification Discovery of ops/tech issues KtrsContractors RTOResponsible Test Organization OTAOperational Test Agency
© 2009 The MITRE Corporation. All rights reserved. DT/OT to IDT/OT Evolution - Future 7 Developmental Test Operational Test Integrated Test OTAs/DOT&E Title 10 confirmation Final certifications Ktrs RTOs OTAs DOT&E JITC Develop/refine requirements Discover/fix ops/tech issues Contractors/RTOs Early technology demo & selection WIN - WIN: Capability in warfighters’ hands ASAP Reduces cost, schedule, and risks
© 2009 The MITRE Corporation. All rights reserved. Agenda/Overview ■What/Why Integrated Testing? ■When Should We Do It? ■What Can/Should Testers Do? ■Summary 8
© 2009 The MITRE Corporation. All rights reserved. Testers Informing Early Decision Making… ■Requirements/specification impact –Disciplined testing mentality helps discovery process; prior to large funding commitment 9
© 2009 The MITRE Corporation. All rights reserved. Integrated Testing - Part of Systems Engineering
© 2009 The MITRE Corporation. All rights reserved. Acquisition Process ICD CT/DT CDDCPD TEMP Operations & Support System Development & Demonstration Design Readiness Review Technology Development Production & Deployment FRP Decision Review LRIP/ IOT&E Concept Refinement Concept Decision A B C OT Test Docs Req’ts TEMP Update T&E 2 yr10 yr8 yr4 yr0 yr Why wait until formal OT to find out you didn’t build the right thing? How far back should we look?
© 2009 The MITRE Corporation. All rights reserved. 12 Integrated Testing Environment Integrated Testing Execution Period
© 2009 The MITRE Corporation. All rights reserved. Agenda/Overview ■What/Why Integrated Testing? ■When Should We Do It? ■What Can/Should Testers Do? ■Summary 13
© 2009 The MITRE Corporation. All rights reserved. 14 During Requirements Development ■Early T&E influence during requirement development : –Ensures requirements are testable and unambiguous ■ Can you readily imagine a verification technique for this requirement? ■ Your requirement may be ambiguous if interpretation is required to design and build –Ensures that the test measures address/answer the Critical Operational Issues –Requirements that are high risk/push the state-of-the-art are identified/addressed Getting the requirements right is a pothole-filled road
© 2009 The MITRE Corporation. All rights reserved. T&E During SOW, RFP, and Contract Development ■T&Eers ensure that test-related scheduling, budgeting, infrastructure, other resources and personnel are included in program’s contractual documents –SOW/RFP, proposal evaluation, contract ■T&Eers help answer the question, “How much and what type of data need to be collected?” –To inform early decisions –Assure continuing involvement ■Test design –Test events /Operational/Mission scenarios –Distributed/Joint/LVC –Statistical sample size determination –Analysis planning 15
© 2009 The MITRE Corporation. All rights reserved. Integrated Test Execution ■Early, continuous IT serves both DT and OT objectives –Provides adequate test time ensuring all test objectives are met –Allows for better/more thorough data collection –Allows PM’s an early look at and time to fix operational issues ■Provides opportunity for assessment of reliability growth –Maintenance activities on early test units yield valuable information ■ Initial maintainability assessments ■ Evolve maintenance approach or design at this relatively inexpensive stage 16 Earlier insight = cheaper & easier fixes
© 2009 The MITRE Corporation. All rights reserved. 17 Opportunities for Integrated Testing ■Integrated DT/OT supported in SOW and RFP ■Integrated DT/OT supported in Test Strategy and Planning documents ■Requirements unambiguous and testable ■DT&E and OT&E personnel working together from Program’s outset ■Contractor and Government DT Data augments OT&E data ■OT&E personnel influence Operational Realism of DT&E scenarios ■Test Planning Working Groups develop common data collection and reporting formats ■The DT&E System Under Test (SUT) reflects Operational System and is operated by typical/actual users ■OT&E personnel participate in DT Testing and/or receive test documentation ■OT&E personnel assist Program Managers to develop “Acceptable Work-arounds” mitigating serious deficiencies
© 2009 The MITRE Corporation. All rights reserved. Recommendations for Integrating DT/OT 1. Build a culture of Mutual Trust and cooperation ■ Demonstrate to PMs OT is not simply “Pass or Fail” ■ Assist PMs in lowering overall program risk by providing SE and T&E involvement early in the development process 2. Develop a “Partnership” not an adversarial relationship between the Test Community and Contractors or PMs ■ Identify ways Integrated DT/OT is of mutual benefit to PM’s, Contractors and OTA’s ■ Demonstrate how early detection of deficiencies reduces overall risk and cost ■ Show how schedule test time can be reduced by integrating DT/OT ■ Assist PM’s in developing and presenting an acceptable case for correcting deficiencies to the MDA
© 2009 The MITRE Corporation. All rights reserved. 3. Encourage adoption of Common data Format/Standards for all Joint and Distributed ■Emphasize need for ability to exchange information in the TEMP as part of Integrated testing ■Include requirement for common data standards/formats in all SE and T&E Instructions and documentation ■DOT&E and OTAs encourage Test Planning Working Groups to develop a common standard for data ■Utilize professional organizations like ITEA and NDIA to support the necessity of common data standards/formats Recommendations for Integrating DT/OT Page 19
© 2009 The MITRE Corporation. All rights reserved. Agenda/Overview ■What/Why Integrated Testing? ■When Should We Do It? ■What Can/Should Testers Do? ■Summary 20
© 2009 The MITRE Corporation. All rights reserved. Integrated Testing Is Critical ■Combines the engineering expertise of the developer and SE with the operational expertise of the user and operational tester ■Ensures that requirements are written correctly ■Ensures all relevant testing is planned and budgeted ■Takes advantage of all available test data to inform acquisition and operational decisions ■Accelerates deployment of mission capability to the warfighter 21
DT&E Cmte - Walking The Line with Title 10 1 Beth Wilson Industry Co-Chair NDIA System Engineering Division, DT&E Committee Test Architect, Raytheon Company.
Verification and Validation — An OSD Perspective — Fred Myers Deputy Director, Test Infrastructure Test Resource Management Center November 4, 2009.
Quality Assurance/Quality Control Plan Evaluation February 16, 2005.
DEVELOPMENTAL TEST & EVALUATION DT&E – From Concept to Combat Integrated Test Process Darlene Mosser-Kerner Developmental Test & Evaluation OUSD(AT&L)/Systems.
Systems Engineering From a Life Cycle Perspective John Groenenboom Director Engineering – Mesa Boeing Rotorcraft Dec 12, 2007.
Lisa Brown and Charles Thomas LAWNET 2002 Taking the Mystery Out of Project Management.
CURRENT STATE OF ACQUISITION, TEST & EVALUATION ENTERPRISE NOVEMBER 2012 PRESENTATION TO INCOSE ENCHANTMENT CHAPTER Charles Pete Adolph.
Collaboration to Meet Future T&E Needs ITEA 14 September Mr. Mike Crisp Deputy Director, Air Warfare Operational Test and Evaluation.
1)List and briefly describe the three project quality management processes. Quality Planning: Identify which quality standards are relevant to project.
1 Dr. Ashraf El-Farghly SECC. 2 Level 3 focus on the organization - Best practices are gathered across the organization. - Processes are tailored depending.
Training Guide Gaining Senior Leadership Support for Continuity of Operations.
Baker Tilly refers to Baker Tilly Virchow Krause, LLP, an independently owned and managed member of Baker Tilly International © 2013 Baker Tilly Virchow.
Effective Contract Management Planning For Performance-Based Contracting.
Workforce Opportunity Services An Overview 1 Dr Art Langer.
GAO’s Cost and Schedule Assessment Guides U.S. Government Accountability Office Applied Research and Methods Cost Engineering Sciences Jason T Lee, Assistant.
PROJECT MANAGEMENT BASICS. CHAPTER 1 - INTRODUCTION 1.1 The PMBOK Guide 1.2 What is a Project? 1.3 What is Project Management? 1.4 Areas of Expertise.
Company LOGO. Company LOGO PE, PMP, PgMP, PME, MCT, PRINCE2 Practitioner.
Acquisition Planning and Adequate Market Research National Oceanic and Atmospheric Administration Acquisition and Grants Office Oversight and Compliance.
Optimize tomorrow today. TM Cost and Affordability approach at Development Planning stage 1.
UNCLASSIFIED DISTRIBUTION STATEMENT A – Cleared for public release by OSR on Sep – SR case number 10-S-3203 Developmental Test & Evaluation Presented.
Certification and Accreditation CS Phase-1: Definition Atif Sultanuddin Raja Chawat Raja Chawat.
U Development of Reliability Program Standard GEIA-STD-0009 SE Summit March 2009 Approved for Public Release; Distribution is unlimited.
GUIDING PRINCIPLES FOR THE FEDERAL ACQUISITION SYSTEM Michael K. Barrett, CPA, CPCM, PMP Barrett Consulting LLC Disclaimer: The views presented are those.
Alpha Contracting Sole Source Negotiations between Government and Industry (13 CFR (c)(2))
Pertemuan 14 Matakuliah: A0214/Audit Sistem Informasi Tahun: 2007.
2 Session Objectives Increase participant understanding of effective financial monitoring based upon risk assessments of sub-grantees Increase participant.
Enterprise Performance Life Cycle (EPLC) Stage Gate Reviews Kevin Lyday & Robin Eggleston February 6, 2009.
11 ADM2372 Management Information Systems (MIS) Chapter 10 – Part I Systems Development Chapter 10 – Part I Systems Development.
CEM515QUALITYASSURANCECEM515QUALITYASSURANCE SUPPLIER PARTNERSHIP.
ODOT’s Public Involvement Process PI and the Project Development Process Minimum PI Requirements.
CHECKPOINTS OF THE PROCESS Three sequences of project checkpoints are used to synchronize stakeholder expectations throughout the lifecycle: 1)Major milestones,
Change Management Overview. 2 Objectives Overview of the change management approach Clarity on how the tools support the change approach Apply the change.
- 1- Just enough. Just in time. Just for you March 24, 2011 Building a solid foundation: A new approach to ERP acquisition March 24, 2011 Mike Reyna Project.
GBA IT Project Management Final Project - Establishment of a Project Management Management Office 10 July, 2003.
EMIS 7307 T&E Part 2 1 Documents in flux. MNS - Mission need statement –Non system specific, a needed capability. Being replaced by Initial Capabilities.
Phase B Exit Criteria Issues Resolved or Addressed Technical Reviews / Readiness Reviews, Audits Initiate Milestone C Program Review Planning Process Initiate.
NDIA SE Division Meeting February 13, Developmental Test and Evaluation Committee Beth Wilson, Raytheon Steve Scukanec, Northrop Grumman Industry.
Outsourcing: Managing Interorganizational Relations CHAPTER TWELVE Student Version Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved.
Software Engineering Process - II 10.1 Unit 10: Integrated Capability Maturity Model (CMMI) Software Engineering Process - II.
Chapter Chapter 13-2 Chapter 13: Developing and Implementing Effective Accounting Information Systems Introduction System Development Life Cycle.
1 Independent Verification and Validation Current Status, Challenges, and Research Opportunities Dan McCaugherty IV&V Program Manager Titan Systems Corporation.
CBIIT Quality Assurance Process Preston Wood NCI CBIIT Government Quality Representative (GQR) January 2014 RS.
1 ME Spring 2015 Systems Engineering, Part II Session 8 5 February 2015 Mr. Larry Hopp, CPL © Copyright 2013.
Privacy Impact Assessment Future Directions TRICARE Management Activity HEALTH AFFAIRS 2009 Data Protection Seminar TMA Privacy Office.
© 2008 Prentice Hall9-1 Introduction to Project Management Chapter 9 Managing Project Risk Information Systems Project Management: A Process and Team Approach,
HOTLINE: The Value of internal Audit at Georgia Tech 1 Department of Internal Auditing.
1 Life Cycle Logistics. 2 The Total Life Cycle Systems Management Initiative TLCSM Definition: TLCSM is the implementation, management, and oversight,
Information Systems Development : Overview. Information systems development practice Concept and role of a systems development methodology Approaches.
© 2009 The MITRE Corporation. All rights Reserved. Evolutionary Strategies for the Development of a SOA-Enabled USMC Enterprise Mohamed Hussein, Ph.D.
1 The Voice of the Industrial Base 1 IBR Guide NDIA PMSC Working Group NDIA PMSC Meeting August 7, 2007 Prepared by; NDIA IBR Working Group August 2007.
© 2017 SlidePlayer.com Inc. All rights reserved.