1. Integrated Survivability Assessment (ISA) for OT&E
Cleared for Public Release
Distribution Unlimited
Integrated Survivability Assessment (ISA) for OT&E
Ron Ketcham
Chairman, JASPO Survivability Assessment Subgroup
Naval Air Warfare Center, Weapons Division
Code D
(760) (DSN ) FAX: (760)
Dave Hall
SURVICE Engineering Company
900E N. Heritage Drive, Suite 1
Ridgecrest, CA 93555
(760) X106; FAX: (760) ; Cell: (760)
The operational test and evaluation (OT&E) community is required to perform an assessment of the effectiveness and suitability of air weapons systems from the standpoint, among others, of survivability. In addition, the OT&E community is required to perform live fire test and evaluation (LFT&E) of the vulnerability of these same air weapons systems. These two assessments are historically done separately, by different test communities, at different times, using different metrics. However, vulnerability of the system is a component of survivability, which makes an overall survivability assessment difficult without incorporating the results of the LFT&E. Combining LFT&E and OT&E issues in an integrated fashion for a true survivability assessment is critical to a valid operational test and evaluation program.

2. ISA Project
Motivation: comprehensive OT&E system survivability evaluations
Tasking from Joint Aircraft Survivability Program Office (JASPO)
At the request of DOT&E
Objectives:
Combine the results of LFT&E and OT&E into an overall assessment of the survivability of a system under test
Adaptable for use in all phases of system development and fielding from Concept Definition through FOT&E
Make use of appropriate M&S and T&E Resources
In response to the DOT&E direction to develop an ISA process outline, the JASPO instituted a project called “Integrated Survivability Assessment”. The plans for FY02 funding of that project (and for a redirected related FY01 project) were to first develop a “checklist” of survivability features and objectives that should be evaluated for any air vehicle system. Second, a hierarchy of survivability metrics was developed for survivability evaluations in combined OT&E and LFT&E; these metrics would be the means to evaluate the checklist. Third, an outline ISA process was described for how those metrics could be measured today, making use of existing (JASPO) modeling and simulation (M&S) and tri-service test range assets. Fourth, three examples of different types of air weapons system acquisition programs were notionally measured using that ISA process outline, to demonstrate the concept. Based on the results of the notional examples, deficiencies were identified in the ISA process, the types of data available to the process, and the M&S available to support the process.
In FY03/04, the JASPO is investigating in more detail the current capabilities of test ranges and the M&S to support the process, and will be coordinating with the Service OT&E agencies on the overall approach. In FY04 JASPO will begin to execute the plan for filling the deficiencies in the ISA process and will begin to execute the ISA process for a representative acquisition program requiring an integrated survivability OT&E and LFT&E program (the primary candidate is the Multi-Mission Maritime Aircraft – MMA). This will be done primarily by supporting the development of a Test and Evaluation Concept Paper for that program, documenting the results, and folding the results into the plans for improving the ISA process.

3. Elements of Survivability
2. Crew Protection
3. Damage
Resistance
4. Signature Reduction
1. Damage
Tolerance
5. Defensive
ECM/IRCM
13. Tactics &
Doctrine
Vulnerability
(LFT&E)
Susceptibility
12. Policy
& ROE
Elements of
Survivability
6. Situational
Awareness
Other
11. Mission
Planning
Outside Factors Affecting Susceptibility
7. Aircraft
Performance
By Survivability we mean all of these factors which affect the ability of a vehicle to successfully operate in it’s tactical environment.
10. Off
Board
Jamming
8. Precision & Standoff Weapons
9. SEAD &
DEAD

4. Tasking
Develop a combined survivability OT&E/LFT&E Checklist to evaluate all aspects of the survivability of air vehicles
Develop metrics to evaluate the checklist
Develop an ISA process for measuring those metrics using current JASPO and DOT&E methodologies, T&E range capabilities and tools
Identify deficiencies in the ISA process (M&S, data, T&E, etc.) and develop detailed plans to mitigate them
Exercise the ISA process for one or more acquisition programs

5. Integrated Survivability Assessment Process
Mission-
Threat
Analysis
EW/EC
Target
Acquisition
Signature
Prediction
Sensor
Models
SAM
AAM
Guns
Lasers
Threat Engagement
Assessment
Vulnerability
Air-to-Air
Combat
Air-to-Ground
Missions
Mission
Survivability
Requirements
Definition
AOA
OT&E/LFT&E
Specification
Compliance
Planning
Susceptibility Assessment

6. Integrated Survivability Assessment
A process to evaluate all aspects of system survivability in a coordinated fashion
Using both M&S and T&E resources where appropriate
Including LFT&E and survivability OT&E issues
Perform survivability OT&E and LFT&E in the context of missions and scenarios for the system under test
By using appropriate “vignettes”
Vignette Matrix identifies potential OT&E/LFT&E test conditions
Provides a framework for developing LFT&E and survivability OT&E test plans
Negotiated between PM, Service OT, DOT&E
Provides a roadmap for use of M&S in support of TEMP
Relates to JSF, AIM-9X experience
Developed example matrices of vignettes representing missions & scenarios for three aircraft types
Helicopter, Transport, TACAIR

7. Example Helicopter Vignettes
Urban
Forest
Desert
AFGHAN
Mountains
SWA
NEA
Somalia
Driving
factors
Close engagement range, hard to find targets
IADS, weather, RF threats, MANPADS, target acq.
Flat terrain, clear weather, CB threat
High altitude,
rough terrain X
Attack Helo
AIR CAV
Takeoff &
Landing
CAS
Battlefield
Interdiction
Air Mobile
Escort
CSAR
Escort**
RECCE*
SCREEN
Target Acq
Air Combat 10 5 30 20
100
Sample
Weights
Missions
Roles
*Most likely mission
**Most stressing mission
Ж = Most
stressing
scenario Ж

8. Example Helicopter Vignette
Close Air Support to troop insertion mission in Operation Anaconda
Combined operation with AH-64, UH-60, CH47, CAS assets
Identified LZ location, mission objectives
Rugged terrain
Uncertain threat situation
Uncertainty in threat means that a number of variations in the vignette should be considered
To determine “robustness”, or sensitivity of survivability results to threat assumptions

9. Principal Survivability Metrics
Mission Level Survivability
Engagement Level Susceptibility
Threat Envelopes (with and w/o CM)
F-Pole, A-Pole, E-Pole
Aircraft Pk/h (or damage given a hit or an intercept)
Threat System Pk Envelopes
Robustness
Vignettes Accomplished: percentage of vignettes where mission can be accomplished considering survivability constraints
Threat Shot Opportunities
Engagement Level Survivability
Expected # casualties given a hit
Crew Survivability
Vulnerability

10. Pk, OT&E, M&S and LFT&E (for Survivability Assessment)
PK/E = PA/E*PT/A*PL/T*PI/L*PF/I*PH/F*PK/H
OT&E
M&S
LFT&E
E = Engagement
A = Acquisition
T = Track
L = Launch
I = Intercept
F = Fuzing
H = Hit
K = Kill

11. Process Guidance and Constraints
Characteristics of System under Test
ISA Process
Create
Vignettes
Applicable
to Program
Generic Vignettes
Program Documents
(Example: TEMP, ORD)
Data Collected from DT&E, LFT&E
(to date and expected)
Develop Test Plans
Assess M&S
Needs
Data Collected from OT&E
(to date and expected)
Select M&S Suite
Best-Suited
for Program
Library of M&S (incl.
Credibility Information)
Conduct T&E
Existing Vulnerability Data
Execute and Collect data from M&S Suite
Collect Data from LFT&E
Collect Data from OT&E
Existing Susceptibility Data
A generalized diagram of the ISA process is shown here. Under the constraints and guidance for the specific assessment being accomplished (detailed on the next chart), an analysis is conducted of the missions and scenarios in which the system under test is expected to operate, the characteristics of the SUT, and program documents describing the system and proposed OT&E and LFT&E; this analysis results in a set of vignettes against which the SUT will be assessed. These vignettes will form the basis of the conditions under which the SUT will be tested.
Data from previous DT&E and LFT&E, any initial OT&E testing accomplished, and and the program documents will be assessed to determine what the needs are for M&S to support the ISA; information on existing M&S, vulnerability data and susceptibility data will determine which M&S are best suited to the program and any M&S improvements required to support the OT&E/LFT&E.
Once all this “up-front” work is accomplished, the tests and any supporting M&S analyses are conducted and evaluated. These analyses then provide values for the survivability metrics chosen under the ISA to measure the survivability of the SUT.
Conduct Analysis
Assess and combine M&S and T&E Results
Process Guidance and Constraints
Survivability Metrics

12. Mission and System Level Test
System Vignettes
&
Survivability Metrics
Test Via
Mission
Operational Test
Force Development/ Experimentation
Tasks (UJTL)
Interoperability Testing
Interactions
Actions
Subsystem Test
ANALYZE
SYNTHESIZE
Components
Component Testing
Technology
Tech Demo
Underlying Figure taken from ITEA Journal, Sept/Oct 2003, “The Test and Evaluation Master Plan: An Operational View”. Daly, Miller, Seglie & Wauer

13. Some Known Deficiencies in ISA Process
General Issues
Model linkages
M&S Validation data availability, coordination with TEMP
Mixed force level assessment (T&E and M&S)
Modeling and Simulation Deficiencies
DECM/IRCM effectiveness
Near field signatures, threat fuze performance
Fire and Explosion
Human performance
C4ISR
Test Range Deficiencies
Missile Miss Distance Measurement
Threat System Variability – system to system variations
Signal Density – not representative on ranges
Semi-Active systems – can’t get realistic test data on ECM effectiveness, etc.
Number of platforms, threats in test, test range size – can’t fully test integrated system
Completeness & fidelity of OT&E data – not generally usable for M&S validation, sometimes not sufficient for test issue resolution

14. ISA Demonstration
Demonstrate ISA process for an actual acquisition program
First candidate: Multi-Mission Maritime Aircraft (MMA)
FY04 Plans:
Develop focused ISA plan for MMA
Develop and coordinate checklist, metrics and vignettes with program office, LFT&E and OT&E agencies
Begin developing database requirements to support ISA assessment

15. Status & Recommendations
Draft ISA process developed
Identified checklist, metrics, models, test data needed
Notional examples worked (Transport, TACAIR, Helicopter)
Identified deficiencies in process, models and data
Draft report reviewed by Tri-Service committee
Developing detailed plans for process improvements
Fleshing out detailed ISA requirements for test data and M&S
Initiating test range capabilities assessment against those requirements
Correlated with ongoing range capabilities improvements
Provides roadmap for both M&S and T&E investment
Recommendations:
Complete and execute ISA process improvement plan
Coordinate with CTEIP and JASPO funding processes
Investigate POM plus-up opportunities
Investigate Training Range capabilities and limitations to support ISA
Develop detailed procedures manuals to supplement TEMP
ISA Demonstration
The initial development of the checklist, metrics and overall ISA process have been completed, including a “notional” run-through of the process for three aircraft types: helicopter, transport aircraft, and a tactical air system. A few more details are given in the “supplemental material”. Informal reviews by the Institute for Defense Analyses, DOT&E, the Joint Technical Coordinating Group on Aircraft Survivability (JTCG/AS) and the Service OT&E organizations may provide for some revision of this initial draft process.

16. Backups

17. Metrics and the Checklist
Links in the Threat Kill Chain
ISA Metrics
Potential Survivability Enhancement Features Along the Kill Chain
Mission Survivability
Missions Accomplished; robustness
All features combine to support mission level survivability
Threat Suppression
Threat Shot opportunities; situational awareness (number, timeliness and accuracy of threats detected)
Tactics, Precision Guided Munitions, mission planning system, low signatures, fighter escort, anti-radiation missiles, self defense weapons
Detection Avoidance
Threat Detection & Acquisition Envelopes
Standoff Weapons, Night-time capability, on-board electronic attack (EA) systems, stand-off EA, low signatures, good target acquisition, terrain following (NOE flight), situational awareness, chaff, threat warning, tactics, mission planning system
Engagement Avoidance
Threat Tracking envelopes; F-Pole, A-Pole, E-Pole; ECM effectiveness
Standoff Weapons, onboard EA, off-board EA, low signatures, good target acquisition, situational awareness, chaff and flares, threat warning, speed and altitude, mission planning system
Threat or Hit avoidance
Threat Intercept Envelopes; ECM/IRCM effectiveness
On-board EA, low signatures, chaff and flares, threat warning, speed and altitude, maneuverability, agility (last ditch maneuver)
Threat or hit tolerance
Threat system Pk envelopes; Aircraft Pk/h; Component Pk/h; VA; list of vulnerable components; Expected # casualties given a hit; hit locations on aircraft
Fire/explosion protection, self-repairing flight controls, redundant and separated hydraulics, multiple engines, no fuel adjacent to air inlets, hydrodynamic ram protection, nonflammable hydraulic fluid, rugged structure, armor