Unmanned Aerial System (UAS) Selection: Validating the Performance Based Measurement (PBM) Battery Presenter: CDR Henry Phillips Military Deputy, Research & Technology Author/PI: Ms. Jennifer Pagan Research Psychologist jennifer.pagan1@navy.mil Co-Authors/Associate Researchers: Mr. Hal Issen Research Psychologist hal.issen@navy.mil Dr. Randy Astwood Senior Research Psychologist randy.astwood@navy.mil

Overview Manned & Unmanned Safety Concerns & Mitigation UAS Selection Challenges Performance Based Measurement Battery NAWCTSD Selection Effort Content Validation Results & Discussion Conclusion

Need Exists to Improve Safety & Mitigate Mishaps UAS Safety Concerns U.S. Military Aircraft and UAS Class A Mishap Rates (Lifetime), 1986–2006* UAV Mishap Causes: Human Factors (≈50%) (Thompson, et al., 2005**; Williams, 2004) Common Issues: Deficient Knowledge, Skills, & Abilities (KSAs) Workload Situational Awareness Decision-making Crew Resource Management Estimates of the percentage of accidents that implicate human error range from 70% to 80% (Wiegmann & Shappell, 2003). In addition, over the past 40 years, the percentage of accidents attributable to human error has increased relative to those attributable to equipment failures (Shappell & Weigmann, 2000) Potential Avoided Cost of Millions of Dollars These data show a mishap rate (i.e., Class A accidents per 100,000 hours of flight) of 20 for Predator, 47 for Hunter (24 since the major reliability improvements in 1996), 88 for Global Hawk, 281 for Pioneer, and 191 for Shadow. For comparison to two manned military aviation mishap rates, the U-2 and F-16 have cumulative Class A mishap rates of 6.8 and 4.1 per 100,000 hours, respectively. Comparing to non-military aircraft, general aviation suffers about 1 Class A mishap per 100,000 hours, regional/commuter airliners about a tenth of that rate, and larger airliners about a hundredth of that rate. Plenty of evidence to indicate that the mishap rates are sig higher for UAV than manned a/c; this graphic provides visual rep e.g., Low end 20 mishaps per 100K flight hours; compared to manned F-16 5-6 100K flight hours; Need Exists to Improve Safety & Mitigate Mishaps *Office of the Secretary of Defense, 2003 **Thompson, Tvaryanas, & Constable, 2005

Mishap Reduction: Lessons Learned From Manned Aviation Naval Aviation Selection Test Battery (ASTB) (11.4% of Variance*) Consists of 6 Cognitive Abilities Tests Reading Skills Test Mathematical Skills Test Mechanical Comprehension Test Spatial Apperception Test Aviation & Nautical Information Test Aviation Supplemental Test Estimated Cost Avoidance of $30M Annually (*Navy Aerospace Medical Institute, 2011) Improved Performance & Training Efficiency Reduced Training Attrition (25%) Improved Safety Field Flight Performance Board Appearance Associated with Low ASTB Scores (Grubb & Phillips, 2011) ASTB: Used to predict success in aviation officer training programs Reduce Training Attrition Attrition reduced from >50% to <25% Improve Training Efficiency Attrition avoidance estimated to save $38M annually Applicants who score better do better in class and flight (through primary) Improve Safety Need metrics and data Ensure the Quality of Fleet Aviators Providing a Comparable Tool for the UAS Community Could Yield Similar Benefits

UAS Selection Challenges Many For Manned Aviation, One For UAS Computer Based Performance Test (CBPT) Only Test Ever Validated for UASs Developed & Validated for Legacy System, Pioneer External Pilots: r²=.86* (Biggerstaff, Blower, Portman, & Chapman, 1998) Internal Pilots: r= .59 (Phillips, Arnold, & Fatolitis, 2003) Highly Predictive But Never Transitioned Standalone System Running On Outdated Operating System No Previous Efforts on Cross Platform Selection Tool Biggerstaff et al., 86% of the variance in performance on instructor evals & flight performance were predicted by the test Criterion: Bi Weekly Instructor Evals & Flight performance Phillips et al (2003) Index score and all score components correlated strongly with training performance and reliably differentiated between attriting and non-attriting students. final average of test performance and flight evaluations for UAV operator training curriculum, Last bullet: Confident in transferable to similar platforms (i.e., Raven,) and other small platforms Objective: Investigate the Utility/Generalizability of a Manned Aviation Validated Selection Tool (i.e., Performance Based Measurement Battery) for Unmanned Systems *Adjusted for Small n

Why the PBM? CBPT Developed for Pioneer, RQ-7 Shadow Replaced Pioneer CBPT (Legacy Pioneer) PBM (Manned Aviation) CBPT Developed for Pioneer, RQ-7 Shadow Replaced Pioneer KSAs Likely to be Similar Other Operationally Similar Platforms Likely to Apply (e.g., Raven, ScanEagle) Subtest Ability Subtest Ability Dichotic Listening Test Auditory Processing Cognitive Processing Dichotic Listening Test Auditory Processing Cognitive Processing Throttle (Vertical Tracking) Test Psychomotor Vertical Tracking Test Psychomotor Stick (2-D Tracking) Test Psychomotor Airplane Tracking (2-D) Test Psychomotor Manikin Test Spatial Ability Cognitive Processing Directional Orientation Test Spatial Ability Cognitive Processing Digit Cancellation Test Cognitive Processing Emergency Scenario General Cognitive Stress Tolerance Rudder (Horizontal Tracking) Test Psychomotor CBPT: 10 Subtests Stick/ATT: Horizontal & Vertical Tracking (2-D) PBM 7 Subtests Emergency Scenario: AttVttEst: General Cognitive Ability, Spatial Ability, Psychomotor Dexterity, Stress Tolerance ATT ATTVTT Multi Tracking Task: ATTVTTDLT EST ATTVTTEST

NAWCTSD Selection Effort Validated UAS Selection Test Validation Process Product Content Construct Criterion 3 Year Effort Underway Exploring Validity of PBM Research Goals Identify Leverage Points Validated Subtests for UAS Identify PBM Gaps Additional Subtests for Future Development Why Reinvent the Wheel? $5M Program to Add PBM to Next Version of ASTB-E Cost Avoidance UAS Cross Platform Analysis: Completed Early this Year 7 Platforms NAMRU D, TSD, NAWCAD

Content Validation: Method Relevant KSAs Expanded Mangos et al 67 KSAs 109 KSAs Used KSA Assessment Tool Sample Item: Identify the degree to which the PBM subtest captured this KSA: Survey Administration PBM Subtests Taken to Provide Context One KSA Assessment Survey Provided following Each Subtest Analysis Method: Two Hurdle Approach Mean Cutoff: 3.5 or Greater Inter-rater Agreement: rwg = .56 or Greater Not at All (0%) Slightly (20%) Partially (40%) Moderately (60%) Strongly (80%) Entirely (100%) 1 2 3 4 5 6 Upon completion of each individual subtest, you will be asked to complete a Knowledge, Skills, Abilities (KSAs) assessment survey. The goal of this exercise is to assess the ability of the Performance Based Measurement Battery to capture KSAs relevant to the Unmanned Aviation Systems (UAS) domain. For each survey, please rate the degree to which you believe the subtest measures each KSA. The 6-point Likert type scale used for the survey is provided below: 1 Not at All: The subtest does not measure this construct at all (0%) 2 Slightly: The subtest measures about 20% of this construct 3 Partially: The subtest measures about 40% of this construct 4 Moderately: This subtest measures about 60% of this construct 5 Strongly: This subtest measures about 80% of this construct 6 Entirely: This subtest measures this construct completely (100%) The first hurdle used a mean cut-off value of 3.5, indicating that the subtest captured at least 50% of the construct. The second hurdle looked at agreement amongst the raters, in which the rwg statistic was calculated.

Content Validation: SME Demographics Participating Organizations Navy Naval Air Warfare Center Training Systems Division, Orlando Naval Medical Research Unit – Dayton Air Force Air Force Personnel Center Air Education & Training Command/Air Force Recruiting Service 9 Subject Matter Experts Experience Type Job Title n Research Psychologist/Researcher 5 Senior Research Psychologist 1 Director/Deputy of Research 2 Experience Type Mean (months) UAS 49 Selection 165.9 Training 60.2

Content Validation: Results Item # KSA DOT DLT VTT ATT VTTATT MTT EST 1 Reasoning Skills X 2 Aviation Principles  20 Spatial Visualization 22 Reaction Time 23 Handling Crisis/Emergency Situations 25 Manual Dexterity 27 Perceptual Speed & Accuracy 34 Control Precision 43 Map Reading 53 Spatial Orientation 56 Mental Rotation 59 Attention Allocation and Control 78 Concentration/Selective Attention 82 Auditory Attention/Localization 85 Rate Control 94 Multilimb Coordination 97 Hand-eye coordination 103 Response Selection Green Checks indicate both criteria met Yellow Xs indicate rwg criteria not met

Result Discussion All Subtests Contain 3+ UAS Relevant KSAs Some Subtests More UAS Valid Than Others? More Validation Work Required to Understand Predictive Ability Limitations Not all SMEs Contained UAS Relevant Domain Knowledge 109 Item KSA Assessment Survey Given 7 Times Fatigue

Future Directions Construct Validation Effort 5 Paper-Based Measures, 1 Computer Based Data Collection 50% Complete Criterion Validation Effort August 2014 Understand Which of the PBM Subtests Predicts Performance for UAS Operators Best Additional Human Factors Questions: Optimizing Performance of Trainees through UAS Manpower, Interface, & Selection (OPTUMIS) Who to Select Training Interface Design

Conclusion Human Causal Factors Attributed to Mishaps Selection Shown to be Beneficial to Manned Aviation Extend to Unmanned 3 Year Effort Underway Exploring Validity of PBM Content Validity 3+ KSAs per Subtest Construct Validation Underway Further Research Necessary

Questions? from 35.8 to 49.0 months, average months of experience in Selection rose from 146.9 to 165.3, and months of experience in Training rose from 52.6 to 60.2.

Research Benefits & Limitation Identify Leverage Points Validated Subtests for UAS Identify PBM Gaps Additional Subtests for Future Development Why Reinvent the Wheel? $5M Program to Add PBM to Next Version of ASTB-E Cost Avoidance Limitations Does Not Answer All Human Factors Questions Who to Select Training Interface Design CONOPs Issues Operators Not in Theater Control of Multiple Disparate UASs Additional Subtests for Future Development Subtests to Discard KSAs Not Captured by the PBM Goal of Research: Provide a Baseline to Guide Future Research Efforts

Mishap Reduction: Lessons Learned From Manned Aviation Naval Aviation Selection Test Battery (ASTB) (11.4% of Variance*) Consists of 6 Cognitive Abilities Tests Reading Skills Test Mathematical Skills Test Mechanical Comprehension Test Spatial Apperception Test Aviation and Nautical Information Test Aviation Supplemental Test Providing a Comparable Tool for the UAS Community Could Yield Similar Benefits

Mishap Reduction: Lessons Learned From Manned Aviation Naval Aviation Selection Test Battery (ASTB) (11.4% of Variance*) Estimated Cost Avoidance of $30M Annually (*Navy Aerospace Medical Institute, 2011) Improved Performance & Training Efficiency Reduced Training Attrition (25%) Improved Safety Field Flight Performance Board Appearance Associated with Low ASTB Scores (Grubb & Phillips, 2011) Providing a Comparable Tool for the UAS Community Could Yield Similar Benefits

 Item # KSA mean r wg DOT 1 3.89 0.448 X 2 3.78 0.590 20 5.33 0.914   DOT 1 Reasoning Skills 3.89 0.448 X 2 Aviation Principles 3.78 0.590  20 Spatial Visualization 5.33 0.914 22 Reaction Time 4.11 0.190 43 Map Reading 0.248 53 Spatial Orientation 4.78 0.676 56 Mental Rotation 5.44 0.819 DLT 3.67 0.829 27 Perceptual Speed and Accuracy 0.276 78 Concentration/Selective Attention 4.67 82 Auditory Attention/Localization 103 Response Selection 0.762

 Item # KSA mean r wg VTT 34 5.44 0.905 85 5.00 0.657 97 4.67 ATT 25   VTT 34 Control Precision 5.44 0.905  85 Rate Control 5.00 0.657 97 Hand-eye coordination 4.67 ATT 25 Manual Dexterity 3.56 0.219 X 5.22 0.848 5.11 0.790 VTTATT 5.33 0.829 59 Attention Allocation and Control 3.78 0.676 4.00 -0.286

 Item # KSA mean r wg MTT 34 5.11 0.619 59 4.56 -0.038 82 4.67 0.571   MTT 34 Control Precision 5.11 0.619  59 Attention Allocation and Control 4.56 -0.038 82 Auditory Attention/Localization 4.67 0.571 85 Rate Control 5.00 0.743 94 Multilimb Coordination 3.56 -0.124 X 97 Hand-eye coordination 4.44 0.305 EST 23 Handling Crisis/Emergency Situations 3.78 0.676 0.829 4.11 103 Response Selection 0.562