Integrated Survivability Assessment (ISA) for OT&E Cleared for Public Release Distribution Unlimited Integrated Survivability Assessment (ISA) for OT&E Ron Ketcham Chairman, JASPO Survivability Assessment Subgroup Naval Air Warfare Center, Weapons Division Code 418100D (760) 939-2363 (DSN 437-2363) FAX: (760) 939-2062 Dave Hall SURVICE Engineering Company 900E N. Heritage Drive, Suite 1 Ridgecrest, CA 93555 (760) 446-4624 X106; FAX: (760) 446-2424; Cell: (760) 382-1618 The operational test and evaluation (OT&E) community is required to perform an assessment of the effectiveness and suitability of air weapons systems from the standpoint, among others, of survivability. In addition, the OT&E community is required to perform live fire test and evaluation (LFT&E) of the vulnerability of these same air weapons systems. These two assessments are historically done separately, by different test communities, at different times, using different metrics. However, vulnerability of the system is a component of survivability, which makes an overall survivability assessment difficult without incorporating the results of the LFT&E. Combining LFT&E and OT&E issues in an integrated fashion for a true survivability assessment is critical to a valid operational test and evaluation program.

ISA Project Motivation: comprehensive OT&E system survivability evaluations Tasking from Joint Aircraft Survivability Program Office (JASPO) At the request of DOT&E Objectives: Combine the results of LFT&E and OT&E into an overall assessment of the survivability of a system under test Adaptable for use in all phases of system development and fielding from Concept Definition through FOT&E Make use of appropriate M&S and T&E Resources In response to the DOT&E direction to develop an ISA process outline, the JASPO instituted a project called “Integrated Survivability Assessment”. The plans for FY02 funding of that project (and for a redirected related FY01 project) were to first develop a “checklist” of survivability features and objectives that should be evaluated for any air vehicle system. Second, a hierarchy of survivability metrics was developed for survivability evaluations in combined OT&E and LFT&E; these metrics would be the means to evaluate the checklist. Third, an outline ISA process was described for how those metrics could be measured today, making use of existing (JASPO) modeling and simulation (M&S) and tri-service test range assets. Fourth, three examples of different types of air weapons system acquisition programs were notionally measured using that ISA process outline, to demonstrate the concept. Based on the results of the notional examples, deficiencies were identified in the ISA process, the types of data available to the process, and the M&S available to support the process. In FY03/04, the JASPO is investigating in more detail the current capabilities of test ranges and the M&S to support the process, and will be coordinating with the Service OT&E agencies on the overall approach. In FY04 JASPO will begin to execute the plan for filling the deficiencies in the ISA process and will begin to execute the ISA process for a representative acquisition program requiring an integrated survivability OT&E and LFT&E program (the primary candidate is the Multi-Mission Maritime Aircraft – MMA). This will be done primarily by supporting the development of a Test and Evaluation Concept Paper for that program, documenting the results, and folding the results into the plans for improving the ISA process.

Elements of Survivability 2. Crew Protection 3. Damage Resistance 4. Signature Reduction 1. Damage Tolerance 5. Defensive ECM/IRCM 13. Tactics & Doctrine Vulnerability (LFT&E) Susceptibility 12. Policy & ROE Elements of Survivability 6. Situational Awareness Other 11. Mission Planning Outside Factors Affecting Susceptibility 7. Aircraft Performance By Survivability we mean all of these factors which affect the ability of a vehicle to successfully operate in it’s tactical environment. 10. Off Board Jamming 8. Precision & Standoff Weapons 9. SEAD & DEAD

Tasking Develop a combined survivability OT&E/LFT&E Checklist to evaluate all aspects of the survivability of air vehicles Develop metrics to evaluate the checklist Develop an ISA process for measuring those metrics using current JASPO and DOT&E methodologies, T&E range capabilities and tools Identify deficiencies in the ISA process (M&S, data, T&E, etc.) and develop detailed plans to mitigate them Exercise the ISA process for one or more acquisition programs

Integrated Survivability Assessment Process Mission- Threat Analysis EW/EC Target Acquisition Signature Prediction Sensor Models SAM AAM Guns Lasers Threat Engagement Assessment Vulnerability Air-to-Air Combat Air-to-Ground Missions Mission Survivability Requirements Definition AOA OT&E/LFT&E Specification Compliance Planning Susceptibility Assessment

Integrated Survivability Assessment A process to evaluate all aspects of system survivability in a coordinated fashion Using both M&S and T&E resources where appropriate Including LFT&E and survivability OT&E issues Perform survivability OT&E and LFT&E in the context of missions and scenarios for the system under test By using appropriate “vignettes” Vignette Matrix identifies potential OT&E/LFT&E test conditions Provides a framework for developing LFT&E and survivability OT&E test plans Negotiated between PM, Service OT, DOT&E Provides a roadmap for use of M&S in support of TEMP Relates to JSF, AIM-9X experience Developed example matrices of vignettes representing missions & scenarios for three aircraft types Helicopter, Transport, TACAIR

Example Helicopter Vignettes Urban Forest Desert AFGHAN Mountains SWA NEA Somalia Driving factors Close engagement range, hard to find targets IADS, weather, RF threats, MANPADS, target acq. Flat terrain, clear weather, CB threat High altitude, rough terrain X Attack Helo AIR CAV Takeoff & Landing CAS Battlefield Interdiction Air Mobile Escort CSAR Escort** RECCE* SCREEN Target Acq Air Combat 10 5 30 20 100 Sample Weights Missions Roles *Most likely mission **Most stressing mission Ж = Most stressing scenario Ж

Example Helicopter Vignette Close Air Support to troop insertion mission in Operation Anaconda Combined operation with AH-64, UH-60, CH47, CAS assets Identified LZ location, mission objectives Rugged terrain Uncertain threat situation Uncertainty in threat means that a number of variations in the vignette should be considered To determine “robustness”, or sensitivity of survivability results to threat assumptions

Principal Survivability Metrics Mission Level Survivability Engagement Level Susceptibility Threat Envelopes (with and w/o CM) F-Pole, A-Pole, E-Pole Aircraft Pk/h (or damage given a hit or an intercept) Threat System Pk Envelopes Robustness Vignettes Accomplished: percentage of vignettes where mission can be accomplished considering survivability constraints Threat Shot Opportunities Engagement Level Survivability Expected # casualties given a hit Crew Survivability Vulnerability

Pk, OT&E, M&S and LFT&E (for Survivability Assessment) PK/E = PA/E*PT/A*PL/T*PI/L*PF/I*PH/F*PK/H OT&E M&S LFT&E E = Engagement A = Acquisition T = Track L = Launch I = Intercept F = Fuzing H = Hit K = Kill

Process Guidance and Constraints Characteristics of System under Test ISA Process Create Vignettes Applicable to Program Generic Vignettes Program Documents (Example: TEMP, ORD) Data Collected from DT&E, LFT&E (to date and expected) Develop Test Plans Assess M&S Needs Data Collected from OT&E (to date and expected) Select M&S Suite Best-Suited for Program Library of M&S (incl. Credibility Information) Conduct T&E Existing Vulnerability Data Execute and Collect data from M&S Suite Collect Data from LFT&E Collect Data from OT&E Existing Susceptibility Data A generalized diagram of the ISA process is shown here. Under the constraints and guidance for the specific assessment being accomplished (detailed on the next chart), an analysis is conducted of the missions and scenarios in which the system under test is expected to operate, the characteristics of the SUT, and program documents describing the system and proposed OT&E and LFT&E; this analysis results in a set of vignettes against which the SUT will be assessed. These vignettes will form the basis of the conditions under which the SUT will be tested. Data from previous DT&E and LFT&E, any initial OT&E testing accomplished, and and the program documents will be assessed to determine what the needs are for M&S to support the ISA; information on existing M&S, vulnerability data and susceptibility data will determine which M&S are best suited to the program and any M&S improvements required to support the OT&E/LFT&E. Once all this “up-front” work is accomplished, the tests and any supporting M&S analyses are conducted and evaluated. These analyses then provide values for the survivability metrics chosen under the ISA to measure the survivability of the SUT. Conduct Analysis Assess and combine M&S and T&E Results Process Guidance and Constraints Survivability Metrics

Mission and System Level Test System Vignettes & Survivability Metrics Test Via Mission Operational Test Force Development/ Experimentation Tasks (UJTL) Interoperability Testing Interactions Actions Subsystem Test ANALYZE SYNTHESIZE Components Component Testing Technology Tech Demo Underlying Figure taken from ITEA Journal, Sept/Oct 2003, “The Test and Evaluation Master Plan: An Operational View”. Daly, Miller, Seglie & Wauer

Some Known Deficiencies in ISA Process General Issues Model linkages M&S Validation data availability, coordination with TEMP Mixed force level assessment (T&E and M&S) Modeling and Simulation Deficiencies DECM/IRCM effectiveness Near field signatures, threat fuze performance Fire and Explosion Human performance C4ISR Test Range Deficiencies Missile Miss Distance Measurement Threat System Variability – system to system variations Signal Density – not representative on ranges Semi-Active systems – can’t get realistic test data on ECM effectiveness, etc. Number of platforms, threats in test, test range size – can’t fully test integrated system Completeness & fidelity of OT&E data – not generally usable for M&S validation, sometimes not sufficient for test issue resolution

ISA Demonstration Demonstrate ISA process for an actual acquisition program First candidate: Multi-Mission Maritime Aircraft (MMA) FY04 Plans: Develop focused ISA plan for MMA Develop and coordinate checklist, metrics and vignettes with program office, LFT&E and OT&E agencies Begin developing database requirements to support ISA assessment

Status & Recommendations Draft ISA process developed Identified checklist, metrics, models, test data needed Notional examples worked (Transport, TACAIR, Helicopter) Identified deficiencies in process, models and data Draft report reviewed by Tri-Service committee Developing detailed plans for process improvements Fleshing out detailed ISA requirements for test data and M&S Initiating test range capabilities assessment against those requirements Correlated with ongoing range capabilities improvements Provides roadmap for both M&S and T&E investment Recommendations: Complete and execute ISA process improvement plan Coordinate with CTEIP and JASPO funding processes Investigate POM plus-up opportunities Investigate Training Range capabilities and limitations to support ISA Develop detailed procedures manuals to supplement TEMP ISA Demonstration The initial development of the checklist, metrics and overall ISA process have been completed, including a “notional” run-through of the process for three aircraft types: helicopter, transport aircraft, and a tactical air system. A few more details are given in the “supplemental material”. Informal reviews by the Institute for Defense Analyses, DOT&E, the Joint Technical Coordinating Group on Aircraft Survivability (JTCG/AS) and the Service OT&E organizations may provide for some revision of this initial draft process.

Backups

Metrics and the Checklist Links in the Threat Kill Chain ISA Metrics Potential Survivability Enhancement Features Along the Kill Chain Mission Survivability Missions Accomplished; robustness All features combine to support mission level survivability Threat Suppression Threat Shot opportunities; situational awareness (number, timeliness and accuracy of threats detected) Tactics, Precision Guided Munitions, mission planning system, low signatures, fighter escort, anti-radiation missiles, self defense weapons Detection Avoidance Threat Detection & Acquisition Envelopes Standoff Weapons, Night-time capability, on-board electronic attack (EA) systems, stand-off EA, low signatures, good target acquisition, terrain following (NOE flight), situational awareness, chaff, threat warning, tactics, mission planning system Engagement Avoidance Threat Tracking envelopes; F-Pole, A-Pole, E-Pole; ECM effectiveness Standoff Weapons, onboard EA, off-board EA, low signatures, good target acquisition, situational awareness, chaff and flares, threat warning, speed and altitude, mission planning system Threat or Hit avoidance Threat Intercept Envelopes; ECM/IRCM effectiveness On-board EA, low signatures, chaff and flares, threat warning, speed and altitude, maneuverability, agility (last ditch maneuver) Threat or hit tolerance Threat system Pk envelopes; Aircraft Pk/h; Component Pk/h; VA; list of vulnerable components; Expected # casualties given a hit; hit locations on aircraft Fire/explosion protection, self-repairing flight controls, redundant and separated hydraulics, multiple engines, no fuel adjacent to air inlets, hydrodynamic ram protection, nonflammable hydraulic fluid, rugged structure, armor