1. Unmanned Aerial System (UAS) Selection: Validating the Performance Based Measurement (PBM) Battery Presenter: CDR Henry Phillips Military Deputy, Research & Technology Author/PI: Ms. Jennifer Pagan Research Psychologist Co-Authors/Associate Researchers:
Mr. Hal Issen Research Psychologist
Dr. Randy Astwood
Senior Research
Psychologist

2. Overview Manned & Unmanned Safety Concerns & Mitigation
UAS Selection Challenges
Performance Based Measurement Battery
NAWCTSD Selection Effort
Content Validation Results & Discussion
Conclusion

3. Need Exists to Improve Safety & Mitigate Mishaps
UAS Safety Concerns
U.S. Military Aircraft and UAS Class A Mishap Rates (Lifetime), 1986–2006*
UAV Mishap Causes:
Human Factors (≈50%) (Thompson, et al., 2005**; Williams, 2004)
Common Issues:
Deficient Knowledge, Skills, & Abilities (KSAs)
Workload
Situational Awareness
Decision-making
Crew Resource Management
Estimates of the percentage of accidents that implicate human error range from 70% to 80% (Wiegmann & Shappell, 2003). In addition, over the past 40 years, the percentage of accidents attributable to human error has increased relative to those attributable to equipment failures (Shappell & Weigmann, 2000)
Potential Avoided Cost of Millions of Dollars
These data show a mishap rate (i.e., Class A accidents per 100,000 hours of
flight) of 20 for Predator, 47 for Hunter (24 since the major reliability improvements in 1996), 88 for
Global Hawk, 281 for Pioneer, and 191 for Shadow. For comparison to two manned military aviation
mishap rates, the U-2 and F-16 have cumulative Class A mishap rates of 6.8 and 4.1 per 100,000 hours,
respectively. Comparing to non-military aircraft, general aviation suffers about 1 Class A mishap per
100,000 hours, regional/commuter airliners about a tenth of that rate, and larger airliners about a
hundredth of that rate.
Plenty of evidence to indicate that the mishap rates are sig higher for UAV than manned a/c; this graphic provides visual rep
e.g., Low end 20 mishaps per 100K flight hours; compared to manned F K flight hours;
Need Exists to Improve Safety & Mitigate Mishaps
*Office of the Secretary of Defense, 2003
**Thompson, Tvaryanas, & Constable, 2005

4. Mishap Reduction: Lessons Learned From Manned Aviation
Naval Aviation Selection Test Battery (ASTB) (11.4% of Variance*)
Consists of 6 Cognitive Abilities Tests
Reading Skills Test
Mathematical Skills Test
Mechanical Comprehension Test
Spatial Apperception Test
Aviation & Nautical Information Test
Aviation Supplemental Test
Estimated Cost Avoidance of $30M Annually (*Navy Aerospace Medical Institute, 2011)
Improved Performance & Training Efficiency
Reduced Training Attrition (25%)
Improved Safety
Field Flight Performance Board Appearance Associated with Low ASTB Scores (Grubb & Phillips, 2011)
ASTB: Used to predict success in aviation officer training programs
Reduce Training Attrition
Attrition reduced from >50% to <25%
Improve Training Efficiency
Attrition avoidance estimated to save $38M annually
Applicants who score better do better in class and flight (through primary)
Improve Safety
Need metrics and data
Ensure the Quality of Fleet Aviators
Providing a Comparable Tool for the UAS Community Could Yield Similar Benefits

5. UAS Selection Challenges
Many For Manned Aviation, One For UAS
Computer Based Performance Test (CBPT) Only Test Ever Validated for UASs
Developed & Validated for Legacy System, Pioneer
External Pilots: r²=.86* (Biggerstaff, Blower, Portman, & Chapman, 1998)
Internal Pilots: r= .59 (Phillips, Arnold, & Fatolitis, 2003)
Highly Predictive But Never Transitioned
Standalone System Running On Outdated Operating System
No Previous Efforts on Cross Platform Selection Tool
Biggerstaff et al., 86% of the variance in performance on instructor evals & flight performance were predicted by the test
Criterion: Bi Weekly Instructor Evals & Flight performance
Phillips et al (2003)
Index score and all score components correlated strongly with
training performance and reliably differentiated between attriting and non-attriting students.
final average of test performance
and flight evaluations for UAV operator training curriculum,
Last bullet: Confident in transferable to similar platforms (i.e., Raven,) and other small platforms
Objective: Investigate the Utility/Generalizability of a Manned Aviation Validated Selection Tool (i.e., Performance Based Measurement Battery) for Unmanned Systems
*Adjusted for Small n

6. Why the PBM? CBPT Developed for Pioneer, RQ-7 Shadow Replaced Pioneer
CBPT (Legacy Pioneer)
PBM (Manned Aviation)
CBPT Developed for Pioneer, RQ-7 Shadow Replaced Pioneer
KSAs Likely to be Similar
Other Operationally Similar Platforms Likely to Apply (e.g., Raven, ScanEagle)
Subtest
Ability
Subtest
Ability
Dichotic Listening Test
Auditory Processing
Cognitive Processing
Dichotic Listening Test
Auditory Processing
Cognitive Processing
Throttle (Vertical Tracking) Test
Psychomotor
Vertical Tracking Test
Psychomotor
Stick (2-D Tracking) Test
Psychomotor
Airplane Tracking (2-D) Test
Psychomotor
Manikin Test
Spatial Ability
Cognitive Processing
Directional Orientation Test
Spatial Ability
Cognitive Processing
Digit Cancellation Test
Cognitive Processing
Emergency Scenario
General Cognitive
Stress Tolerance
Rudder (Horizontal Tracking) Test
Psychomotor
CBPT: 10 Subtests
Stick/ATT: Horizontal & Vertical Tracking (2-D)
PBM 7 Subtests
Emergency Scenario: AttVttEst: General Cognitive Ability, Spatial Ability, Psychomotor Dexterity, Stress Tolerance
ATT
ATTVTT
Multi Tracking Task: ATTVTTDLT
EST ATTVTTEST

7. NAWCTSD Selection Effort
Validated UAS Selection Test
Validation Process
Product
Content
Construct
Criterion
3 Year Effort Underway Exploring Validity of PBM
Research Goals
Identify Leverage Points
Validated Subtests for UAS
Identify PBM Gaps
Additional Subtests for Future Development
Why Reinvent the Wheel?
$5M Program to Add PBM to Next Version of ASTB-E
Cost Avoidance
UAS Cross Platform Analysis:
Completed Early this Year
7 Platforms
NAMRU D, TSD, NAWCAD

8. Content Validation: Method
Relevant KSAs
Expanded Mangos et al 67 KSAs
109 KSAs Used
KSA Assessment Tool Sample Item:
Identify the degree to which the PBM subtest captured this KSA:
Survey Administration
PBM Subtests Taken to Provide Context
One KSA Assessment Survey Provided following Each Subtest
Analysis Method: Two Hurdle Approach
Mean Cutoff: 3.5 or Greater
Inter-rater Agreement: rwg = .56 or Greater
Not at All (0%)
Slightly (20%)
Partially (40%)
Moderately (60%)
Strongly (80%)
Entirely (100%) 1 2 3 4 5 6
Upon completion of each individual subtest, you will be asked to complete a Knowledge, Skills, Abilities (KSAs) assessment survey. The goal of this exercise is to assess the ability of the Performance Based Measurement Battery to capture KSAs relevant to the Unmanned Aviation Systems (UAS) domain. For each survey, please rate the degree to which you believe the subtest measures each KSA. The 6-point Likert type scale used for the survey is provided below:
1 Not at All: The subtest does not measure this construct at all (0%)
2 Slightly: The subtest measures about 20% of this construct
3 Partially: The subtest measures about 40% of this construct
4 Moderately: This subtest measures about 60% of this construct
5 Strongly: This subtest measures about 80% of this construct
6 Entirely: This subtest measures this construct completely (100%)
The first hurdle used a mean cut-off value of 3.5, indicating that the subtest captured at least 50% of the construct. The second hurdle looked at agreement amongst the raters, in which the rwg statistic was calculated.

9. Content Validation: SME Demographics
Participating Organizations
Navy
Naval Air Warfare Center Training Systems Division, Orlando
Naval Medical Research Unit – Dayton
Air Force
Air Force Personnel Center
Air Education & Training Command/Air Force Recruiting Service
9 Subject Matter Experts
Experience Type
Job Title n
Research Psychologist/Researcher 5
Senior Research Psychologist 1
Director/Deputy of Research 2
Experience Type
Mean (months)
UAS 49
Selection
165.9
Training
60.2

10. Content Validation: Results
Item #
KSA
DOT
DLT
VTT
ATT
VTTATT
MTT
EST 1
Reasoning Skills X 2
Aviation Principles  20
Spatial Visualization 22
Reaction Time 23
Handling Crisis/Emergency Situations 25
Manual Dexterity 27
Perceptual Speed & Accuracy 34
Control Precision 43
Map Reading 53
Spatial Orientation 56
Mental Rotation 59
Attention Allocation and Control 78
Concentration/Selective Attention 82
Auditory Attention/Localization 85
Rate Control 94
Multilimb Coordination 97
Hand-eye coordination
103
Response Selection
Green Checks indicate both criteria met
Yellow Xs indicate rwg criteria not met

11. Result Discussion All Subtests Contain 3+ UAS Relevant KSAs
Some Subtests More UAS Valid Than Others?
More Validation Work Required to Understand Predictive Ability
Limitations
Not all SMEs Contained UAS Relevant Domain Knowledge
109 Item KSA Assessment Survey Given 7 Times
Fatigue

12. Future Directions Construct Validation Effort
5 Paper-Based Measures, 1 Computer Based
Data Collection 50% Complete
Criterion Validation Effort
August 2014
Understand Which of the PBM Subtests Predicts Performance for UAS Operators Best
Additional Human Factors Questions: Optimizing Performance of Trainees through UAS Manpower, Interface, & Selection (OPTUMIS)
Who to Select
Training
Interface Design

13. Conclusion Human Causal Factors Attributed to Mishaps
Selection Shown to be Beneficial to Manned Aviation
Extend to Unmanned
3 Year Effort Underway Exploring Validity of PBM
Content Validity
3+ KSAs per Subtest
Construct Validation Underway
Further Research Necessary

14. Questions?
from 35.8 to 49.0 months, average months of experience in Selection rose from to 165.3, and months of experience in Training rose from 52.6 to 60.2.

15. Research Benefits & Limitation
Identify Leverage Points
Validated Subtests for UAS
Identify PBM Gaps
Additional Subtests for Future Development
Why Reinvent the Wheel?
$5M Program to Add PBM to Next Version of ASTB-E
Cost Avoidance
Limitations
Does Not Answer All Human Factors Questions
Who to Select
Training
Interface Design
CONOPs Issues
Operators Not in Theater
Control of Multiple Disparate UASs
Additional Subtests for Future Development
Subtests to Discard
KSAs Not Captured by the PBM
Goal of Research: Provide a Baseline to Guide Future Research Efforts

16. Mishap Reduction: Lessons Learned From Manned Aviation
Naval Aviation Selection Test Battery (ASTB) (11.4% of Variance*)
Consists of 6 Cognitive Abilities Tests
Reading Skills Test
Mathematical Skills Test
Mechanical Comprehension Test
Spatial Apperception Test
Aviation and Nautical Information Test
Aviation Supplemental Test
Providing a Comparable Tool for the UAS Community Could Yield Similar Benefits

17. Mishap Reduction: Lessons Learned From Manned Aviation
Naval Aviation Selection Test Battery (ASTB) (11.4% of Variance*)
Estimated Cost Avoidance of $30M Annually (*Navy Aerospace Medical Institute, 2011)
Improved Performance & Training Efficiency
Reduced Training Attrition (25%)
Improved Safety
Field Flight Performance Board Appearance Associated with Low ASTB Scores (Grubb & Phillips, 2011)
Providing a Comparable Tool for the UAS Community Could Yield Similar Benefits

18.  Item # KSA mean r wg DOT 1 3.89 0.448 X 2 3.78 0.590 20 5.33 0.914
DOT 1
Reasoning Skills
3.89
0.448 X 2
Aviation Principles
3.78
0.590  20
Spatial Visualization
5.33
0.914 22
Reaction Time
4.11
0.190 43
Map Reading
0.248 53
Spatial Orientation
4.78
0.676 56
Mental Rotation
5.44
0.819
DLT
3.67
0.829 27
Perceptual Speed and Accuracy
0.276 78
Concentration/Selective Attention
4.67 82
Auditory Attention/Localization
103
Response Selection
0.762

19.  Item # KSA mean r wg VTT 34 5.44 0.905 85 5.00 0.657 97 4.67 ATT 25
VTT 34
Control Precision
5.44
0.905  85
Rate Control
5.00
0.657 97
Hand-eye coordination
4.67
ATT 25
Manual Dexterity
3.56
0.219 X
5.22
0.848
5.11
0.790
VTTATT
5.33
0.829 59
Attention Allocation and Control
3.78
0.676
4.00
-0.286

20.  Item # KSA mean r wg MTT 34 5.11 0.619 59 4.56 -0.038 82 4.67 0.571
MTT 34
Control Precision
5.11
0.619  59
Attention Allocation and Control
4.56
-0.038 82
Auditory Attention/Localization
4.67
0.571 85
Rate Control
5.00
0.743 94
Multilimb Coordination
3.56
-0.124 X 97
Hand-eye coordination
4.44
0.305
EST 23
Handling Crisis/Emergency Situations
3.78
0.676
0.829
4.11
103
Response Selection
0.562