1. How Do You Test an IT Service? Pete Green Tom Frey

2. Purpose Explore evolving influence of IT systems and services on weapons systems Explore evolving influence of IT systems and services on weapons systems Increasing complexity of testing Increasing complexity of testing Our experience foundation is in NMCI Our experience foundation is in NMCI Foci are the test processes Foci are the test processes

3. How Did We Get Here? JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN TESP Input Delivered Veridian sole sourced By EDS to conduct NMCI CT&E BSA conducted at 4 test sites Master Test Plan Development CT&E 1 Testing Conducted CT&E 3 Testing CT&E 2 Testing Master Test Plan Completed & Delivered Interoperability Testing & Monitoring Plan Delivery Contract SLAs mapped to COIs NMCI Contract Awarded to EDS NMCI Designated DOT&E Oversight Program DASN Directs TESP for NMCI 200120022000

4. Rolling In Compressed Schedule Compressed Schedule No Traditional Program Documentation No Traditional Program Documentation Our Response Based on 4-step Test Process Our Response Based on 4-step Test Process Build Up (Crawl, Walk, Run) Build Up (Crawl, Walk, Run) Concentric Testing Concentric Testing Familiar Fundamentals Familiar Fundamentals Faces Faces Processes Processes

5. Test Approach BSA BSA Phased Testing Phased Testing CTE1 - DT (lab, bench) CTE1 - DT (lab, bench) CTE2 - DT (networks) CTE2 - DT (networks) CTE3 - DT/OT (operational vignettes) CTE3 - DT/OT (operational vignettes) OPEVAL - OT OPEVAL - OT SLA based - derived CTPs, COIs, SLA based - derived CTPs, COIs,

6. Test Approach

8. IT Testing Similar to Early Aviation Testing Similar to Early Aviation Testing How fast can you go? How fast can you go? How high can you go? How high can you go? OSI layer 1-3 centric OSI layer 1-3 centric Parameter focus Parameter focus

9. IT T&E and the OSI 7-Layer Model Where We need to be Mission Based Present Commercial IT Testing Parametric 76543217654321 Is This System Effective and Suitable? How Fast Can This Car Go? Physical Data Link Network Transport Session Presentation Application

10. Product Line Maturity 1950 1960 1970 1980 1990 2000 Systems Maturity Aircraft Avionics Software Networks

11. C4I Testing Success of testing tied to maturity of program execution Success of testing tied to maturity of program execution OTRR OTRR Schedule Constraints Schedule Constraints IP systems generate lots of data IP systems generate lots of data Program Execution curve is fractal Program Execution curve is fractal Processes on critical path provide 1st order approximation Processes on critical path provide 1st order approximation Use of surveys Use of surveys A o A o

12. Testing and Program Execution

13. Test Issues NMCI equal to or better than legacy networks NMCI equal to or better than legacy networks Critical path processes Critical path processes Seat cutover Seat cutover Legacy applications Legacy applications SLAs to COIs not a reversible function SLAs to COIs not a reversible function Operationalizing the metrics Operationalizing the metrics Early and combined CT/OT to minimize risk Early and combined CT/OT to minimize risk

14. Baseline System Assessment Innovative contract vehicle Innovative contract vehicle Lack of traditional documentation Lack of traditional documentation Needed to measure the performance of the legacy systems for comparison with NMCI Needed to measure the performance of the legacy systems for comparison with NMCI

15. BSA Tools AMP AMP NetIQ NetIQ MIB scripts MIB scripts Help Desk Logs Help Desk Logs BelManage BelManage Surveys Surveys End user End user CIO and IT professionals CIO and IT professionals Information Assurance Information Assurance Use same tools and processes for OPEVAL Use same tools and processes for OPEVAL

16. BSA Lessons Learned Configuration management is historical Configuration management is historical Good management makes an immediate impact Good management makes an immediate impact Firewalls and IDS Firewalls and IDS Routers Routers Connectivity Connectivity Legacy applications Legacy applications

17. TESP Process Similar to a TEMP Similar to a TEMP More responsive More responsive If it was not necessary, it was not included If it was not necessary, it was not included If it was necessary, it was tailored to NMCI If it was necessary, it was tailored to NMCI From blank sheet to approval in under 11 months From blank sheet to approval in under 11 months Represented a strategy for success for NMCI Represented a strategy for success for NMCI

18. The State of IT Testing Today Operational assessments of interoperability and information assurance in fielded systems Operational assessments of interoperability and information assurance in fielded systems Inverse correlation of interoperability and information assurance Inverse correlation of interoperability and information assurance Mission Thread Analysis Mission Thread Analysis Blue, Green, and Red Teams Blue, Green, and Red Teams Common Toolkits Common Toolkits Information Operations Threat Capabilities Summaries Information Operations Threat Capabilities Summaries Feedback and feed forward Feedback and feed forward

19. Interoperability vs. Information Assurance: A Complex Challenge Interoperability (Iop) Information Assurance (IA) Iop(a x ) = b y IA An Enterprise System Needs to Balance Iop and IA

20. Testing Frameworks Device Testing System Testing System of Systems Testing Work Flow Threads Mission-Focused Testing A Z

21. Example Mission Thread (ATO Distribution) BCD PLNS WS GCC Prioritized Target List KCOIC 1 TWM & CTB-AM Server KCOIC CTAPS WS HTACC CTAPS APS Server CTAPS ADS Server HTACC CTAPS WS RELROK WAN Integrated Tasking Order 7 KCOIC CTAPS WS KCOIC GCCS-K C2/COP WS CTAPS ITODB Server RELROK WAN ROK AF (Suwon) CTAPS TGT WS 5 KUNSAN CTAPS TGT WS 6 Training Facility GCCS-K Web Server RELROK WAN CP TANGO GCCS-K C2/COP WS 8 RELROK WAN 23 4 2 ACC Target Nomination ITO Sent to TL # 1,6

22. Level of Perf Situation Conditions Joint Mission Areas UJTL Assigned CoCom Missions CoCom JMETs Lessons Learned CCTI Strategies Commanders Training Guidance Exercise Objectives Joint Doctrine TT&Ps Training Objectives Training Objectives BLUFOR OPLAN OPFOR Campaign Control Plan Scenario MSEL Collection Management Plan Integrated Architectures C4ISP Exercise Comm/Sys Architecture Mission Functional Threads Mission Functional Threads Execution Assessment Plan Assessment Plan Performance Data Performance Analysis After Action Review System SMEs Mission SMEs Operational Testers MISSION ISSUES EXERCISE ISSUES SYSTEM ISSUES PERFORMANCE ASSESSMENT Standards Assessment Methodology Flow JTS

23. QUESTIONS?