ATCS Selection background u Previous Selection System l Written aptitude test battery l Academy screening (9 weeks) l On-the-job training u Issues l Test utility after 12-15 years use l Academy screening cost & fairness
Air Traffic Selection and Training (AT-SAT) Validation Research u Integrated validation study l Job-performance criterion measures l 8 new computerized predictor tests (6 hrs) l Concurrent validation study on en route controllers
Order Boxes Belt ABelt BBelt CBelt D Quality Control Conveyor Belt Order Boxes Area Loading Area Quality Control Area Box Storage Area Available Zone Letter Factory Test
Order Boxes Belt ABelt BBelt CBelt D Quality Control A C D P B
Which boxes should be in the loading area in order to correctly place all the letters on the belts? 1. One purple box and one orange box 2. One green box, one purple box, and one orange box 3. Two purple boxes, one green box, and one orange box 4. Two green boxes, one orange box, and one purple box Letter Factory Test - Situational Awareness Questions
Angles Test This test includes two different types of questions: The first presents a picture of an angle and asks you the measure of that angle in degrees (From 1 to 360) What is the measure of this angle? 2) 10°1) 90°3) 125°4) 190°
This test includes two different types of questions: The second provides you with the measure of an angle (from 1 to 360 degrees). It then asks you to select an angle that represents that measure. Which of the following represents a 10 degree angle? 1) 2) 3) 4)
Experience Questionnaire 1- Definitely True 2- Somewhat True 3- Neither True nor False 4- Somewhat False 5- Definitely False Your emotions have sometimes prevented you from solving a difficult problem.
Applied Math The distance from point A to point B is 560 miles. If the aircraft left point A at 8:00 Zulu, and flew at 400 kts, what time would the aircraft cross point B? A. 8:56 B. 9:02 C. 9:24 D. 10:02
Analogies Test Water: Liquid Ice: ? Gas Cube Solid Oxygen Freeze (1) (2) (3) (4) (5)
Plane Icon M2B Exit/Airport B Altitude Level 2 Speed Medium Heading
Design Elements u Environment: airspace sector with exits, airports, planes, and controls u Events: planes appear, move, and disappear u Actions: user clicks controls to control planes. u Rules: eight rules related to speed, altitude, separation, etc.
Display of air sector: four exits two airports planes Plane controls Display of time remaining Environment
Rule violations Separation errors Crashes Elapsed Time Record the Outcomes
N of violations of each rule N of separation errors N of crashes N of successful landings Duration of delays Types of Computed Scores
N of crashes & separation errors N of procedural errors Percent of successful flights Total delay time (handoff & en route) Á Priori Rational Scales
Moved flyovers based on data Standardize scores before combining and weighting Rescale scores if useful: reverse fix skewness make scores more sensitive to differences at higher levels Adjust Scales and Scores
Goals u Find best set of tests for the final battery u Determine scale weights u Measure validity: How well test battery predicts job performance
Decisions to be Made u Which predictors to keep u How to weight predictors
Decision 1: Choosing Predictors u Phase I- Evaluate predictors according to: l simple validity l incremental validity l fairness l group differences l test administration time u Predictors chosen by group consensus
Decision 1: Choosing Predictors u Phase 2: Optimal weighting algorithm l iterated regression l negative weights set to zero l maximizes R 2 while minimizing differences in group means, slopes, and intercepts u Many runs done while varying the relative importance of R 2 and group differences (10 parameters)
Decision 2: How to Weight Predictors u Alternatives Considered l regression weights l validity weights l optimal weights l unit weights u Chosen Weighting Method l mean of validity and optimal weights
Weighting Results: Validity WeightsValidity Shrunken Validity African- American d-score Regression.691.666-.85 Combined.682>.663-.81 Validity.664>.644-.88 Unit.604 -.92 Optimal.631.603-.55 Notes. N = 1029, Regression uses all 35 scales, others use the 26 scales in the final battery.
Conclusions: Optimal Weights u An optimal weighting algorithm can balance the many considerations in selecting and weighting predictors u Optimal weighting tends to exclude too many predictors u Optimal weighting used in conjunction with another weighting method can perform very well
Conclusions: Validity Weights u In the current study, validity weights performed very well, especially when combined with optimal weights u Research is needed to compare the shrinkage (due to overfitting and predictor selection) and R 2 of validity weights to other weighting methods
Estimating Shrinkage in R u Shrinkage formula has shortcomings u Shrinkage formula corrects for overfitting but not predictor selection u Validity weighting - only selection shrinkage u Optimal weighting - shrinkage is severe because of extra parameters
Conclusions: Shrinkage u Research is needed to help estimate shrinkage of R (due to overfitting and predictor selection) using validity weights and optimal weights. u Research needed to compare validity weights, optimal weights, and regression weights in terms of shrunken R 2 (due to overfitting and predictor selection)
Regression Method u Set cut score on criterion u Compute corresponding score on predictor based on regression line u Works well when R-squared is high and good criterion is available
Criterion Cut Point Options u Cut score set at 5% (for example) of current incumbents distribution u Set cut score such that mean expected performance of candidates passing is at 60 th percentile (for example) of current incumbents u Set score at anchor on ratings scale (e.g., acceptable.
Cut Point Considerations u % of applicants passing u % of incumbents passing u % of each minority passing u adverse impact ratio u If low pass rate: l need many applicants, or l recruiting must target high-quality applicants