1. Integrating Developmental and Operational Testing
Dr. William (Dave) Bell
2 December 2009

2. Agenda/Overview
What/Why Integrated Testing?
When Should We Do It?
What Can/Should Testers Do?
Summary

3. Recent DoD Integrated Testing Policy
T&E expertise must be brought to bear at the beginning
of the system life cycle to provide earlier learning about
the strengths and weaknesses of the system under development
Integrated testing is the collaborative planning and collaborative execution
of test phases and events to provide shared data in support of independent
analysis, evaluation and reporting by all stakeholders particularly the
development (both contractor and government) and operational test and
evaluation communities

4. Integrated Testing Defined
Defense Acquisition Guidebook, Chapter 9
Whenever feasible, DT&E and OT&E events should be combined, if that supports technical and operational test objectives to gain the optimum amount of testing benefit for reasonable cost and time. The user community should be involved early in test planning to ensure the statement of desired capabilities is interpreted correctly and tested realistically. Certain events can be organized to provide information useful to developmental and operational evaluators and lend themselves to the DT and OT approach. The concept is to conduct a single combined test program that produces credible, qualitative and quantitative information that can be used to address developmental and operational issues. Examples of this approach include: combined DT/OT events or piggybacking an operational assessment onto a developmental test. Likewise, developmental testing data requirements can be accommodated by an operational test. This approach can reduce the time and expense of conducting dedicated OT events that replicate DT events, or vice versa, yet still provide adequate technical risk reduction. The developmental and operational testers can develop a test management structure to share control of the combined events. Combined DT and OT events and test data requirements must be identified early to prevent unnecessary duplication and to control costs. It is important that neither the DT&E nor OT&E objectives are compromised in designing combined events.
Combined DT & OT events
support technical and operational objectives
User community involved early
to ensure desired capabilities
interpreted correctly and tested realistically

5. DT/OT to IDT/OT Evolution - Past
Developmental
Test
Operational
Test
Contractors
Responsible Test
Organizations
Find & fix technical
problems
Operational Test
Agencies
Director, OT&E
Discover ops issues
& confirmation
Certification

6. DT/OT to IDT/OT Evolution - Present
Developmental
Test
Operational
Test
Integrated
Test
Ktrs
RTOs
Find & fix technical
problems
Certification
OTAs
DOT&E
Finish discovery &
perform confirmation
Ktrs
RTOs
OTAs
DOT&E
Certification
Discovery
of ops/tech
issues
Ktrs Contractors
RTO Responsible Test Organization
OTA Operational Test Agency

7. DT/OT to IDT/OT Evolution - Future
Developmental
Test
Integrated
Test
Operational
Test
Ktrs
RTOs
OTAs
DOT&E
JITC
Develop/refine
requirements
Discover/fix
ops/tech
issues
OTAs/DOT&E
Title 10 confirmation
Final certifications
Contractors/RTOs
Early technology
demo & selection
WIN - WIN:
Capability in warfighters’ hands ASAP
Reduces cost, schedule, and risks

8. Agenda/Overview
What/Why Integrated Testing?
When Should We Do It?
What Can/Should Testers Do?
Summary

9. Testers Informing Early Decision Making…
Requirements/specification impact
Disciplined testing mentality helps discovery process; prior to large funding commitment

10. Integrated Testing - Part of Systems Engineering

11. Production & Deployment
Acquisition Process
Operations &
Support
System Development
& Demonstration
Design
Readiness
Review
Technology
Development
Production & Deployment
FRP Decision Review
LRIP/
IOT&E
Concept Refinement
Concept
Decision A B C
ICD
CDD
CPD
Why wait until formal OT to find out you didn’t build the right thing?
Req’ts
TEMP
TEMP Update
Test Docs
CT/DT OT
T&E
How far back should we look?
0 yr
2 yr
4 yr
8 yr
10 yr

12. Integrated Testing Environment• • •
Integrated Testing Execution Period

13. Agenda/Overview
What/Why Integrated Testing?
When Should We Do It?
What Can/Should Testers Do?
Summary

14. During Requirements Development
Early T&E influence during requirement development :
Ensures requirements are testable and unambiguous
Can you readily imagine a verification technique for this requirement?
Your requirement may be ambiguous if interpretation is required to design and build
Ensures that the test measures address/answer the Critical Operational Issues
Requirements that are high risk/push the state-of-the-art are identified/addressed
Getting the requirements right is
a pothole-filled road 14

15. T&E During SOW, RFP, and Contract Development
T&Eers ensure that test-related scheduling, budgeting, infrastructure, other resources and personnel are included in program’s contractual documents
SOW/RFP, proposal evaluation, contract
T&Eers help answer the question, “How much and what type of data need to be collected?”
To inform early decisions
Assure continuing involvement
Test design
Test events /Operational/Mission scenarios
Distributed/Joint/LVC
Statistical sample size determination
Analysis planning

16. Integrated Test Execution
Early, continuous IT serves both DT and OT objectives
Provides adequate test time ensuring all test objectives are met
Allows for better/more thorough data collection
Allows PM’s an early look at and time to fix operational issues
Provides opportunity for assessment of reliability growth
Maintenance activities on early test units yield valuable information
Initial maintainability assessments
Evolve maintenance approach or design at this relatively inexpensive stage
Earlier insight = cheaper & easier fixes

17. Opportunities for Integrated Testing
Integrated DT/OT supported in SOW and RFP
Integrated DT/OT supported in Test Strategy and Planning documents
Requirements unambiguous and testable
DT&E and OT&E personnel working together from Program’s outset
Contractor and Government DT Data augments OT&E data
OT&E personnel influence Operational Realism of DT&E scenarios
Test Planning Working Groups develop common data collection and reporting formats
The DT&E System Under Test (SUT) reflects Operational System and is operated by typical/actual users
OT&E personnel participate in DT Testing and/or receive test documentation
OT&E personnel assist Program Managers to develop “Acceptable Work-arounds” mitigating serious deficiencies

18. Recommendations for Integrating DT/OT
1. Build a culture of Mutual Trust and cooperation
Demonstrate to PMs OT is not simply “Pass or Fail”
Assist PMs in lowering overall program risk by providing SE and T&E involvement early in the development process
2. Develop a “Partnership” not an adversarial relationship between the Test Community and Contractors or PMs
Identify ways Integrated DT/OT is of mutual benefit to PM’s, Contractors and OTA’s
Demonstrate how early detection of deficiencies reduces overall risk and cost
Show how schedule test time can be reduced by integrating DT/OT
Assist PM’s in developing and presenting an acceptable case for correcting deficiencies to the MDA

19. Recommendations for Integrating DT/OT
3. Encourage adoption of Common data Format/Standards for all Joint and Distributed
Emphasize need for ability to exchange information in the TEMP as part of Integrated testing
Include requirement for common data standards/formats in all SE and T&E Instructions and documentation
DOT&E and OTAs encourage Test Planning Working Groups to develop a common standard for data
Utilize professional organizations like ITEA and NDIA to support the necessity of common data standards/formats

20. Agenda/Overview
What/Why Integrated Testing?
When Should We Do It?
What Can/Should Testers Do?
Summary

21. Integrated Testing Is Critical
Combines the engineering expertise of the developer and SE with the operational expertise of the user and operational tester
Ensures that requirements are written correctly
Ensures all relevant testing is planned and budgeted
Takes advantage of all available test data to inform acquisition and operational decisions
Accelerates deployment of mission capability to the warfighter