1. AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)

2. Topics Today u Background of AT-SAT u Overview of AT-SAT tests u Air Traffic Scenarios Test u Score weighting methods u Setting the cut score

3. Background of AT-SAT

4. ATCS Selection background u Previous Selection System l Written aptitude test battery l Academy screening (9 weeks) l On-the-job training u Issues l Test utility after 12-15 years use l Academy screening cost & fairness

5. Air Traffic Selection and Training (AT-SAT) Validation Research u Integrated validation study l Job-performance criterion measures l 8 new computerized predictor tests (6 hrs) l Concurrent validation study on en route controllers

6. Overview of AT-SAT Tests

7. Dial Reading Letter Factory AT Scenario Angles EQ Applied Math Analogies Scan

8. Dial Reading Test Temp. 1) 22 2) 24 3) 28 4) 30 5) 45

9. Order Boxes Belt ABelt BBelt CBelt D Quality Control Conveyor Belt Order Boxes Area Loading Area Quality Control Area Box Storage Area Available Zone Letter Factory Test

10. Order Boxes Belt ABelt BBelt CBelt D Quality Control A C D P B

11. Which boxes should be in the loading area in order to correctly place all the letters on the belts? 1. One purple box and one orange box 2. One green box, one purple box, and one orange box 3. Two purple boxes, one green box, and one orange box 4. Two green boxes, one orange box, and one purple box Letter Factory Test - Situational Awareness Questions

12. Air Traffic Scenarios Test

13. Angles Test This test includes two different types of questions: The first presents a picture of an angle and asks you the measure of that angle in degrees (From 1 to 360) What is the measure of this angle? 2) 10°1) 90°3) 125°4) 190°

14. This test includes two different types of questions: The second provides you with the measure of an angle (from 1 to 360 degrees). It then asks you to select an angle that represents that measure. Which of the following represents a 10 degree angle? 1) 2) 3) 4)

15. Experience Questionnaire 1- Definitely True 2- Somewhat True 3- Neither True nor False 4- Somewhat False 5- Definitely False Your emotions have sometimes prevented you from solving a difficult problem.

16. Applied Math The distance from point A to point B is 560 miles. If the aircraft left point A at 8:00 Zulu, and flew at 400 kts, what time would the aircraft cross point B? A. 8:56 B. 9:02 C. 9:24 D. 10:02

17. Analogies Test Water: Liquid Ice: ? Gas Cube Solid Oxygen Freeze (1) (2) (3) (4) (5)

18. Visual Analogies: : : (1) (2) (3) (4) (5)

19. V41 610 Type the identification numbers contained in the data block with lower line numbers falling beyond the range (360-710): B12 810 P45 580 T65 120 Y85 710 K23 250 F75 560 C20 390 Scan Test

20. Air Traffic Scenarios Test

21. Simulation Overview u Description of simulation u Design considerations u Scoring u Instructions and practice

22. ATST: Description of Simulation

23. Display of airspace sector Control several planes Land at two airports Exit at four locations AT Scenarios in a Nutshell

24. F2e F4D S1 A F3f F4B M4C

25. Plane Icon M2B Exit/Airport B Altitude Level 2 Speed Medium Heading

26. Design Elements u Environment: airspace sector with exits, airports, planes, and controls u Events: planes appear, move, and disappear u Actions: user clicks controls to control planes. u Rules: eight rules related to speed, altitude, separation, etc.

27. Display of air sector: four exits two airports planes Plane controls Display of time remaining Environment

28. Direction – 8 headings Speed - 3 levels Altitude - 4 levels Accept handoff Actions: Plane Controls

29. New planes appear Planes move Planes disappear: land, exit, or crash Events

30. Land at slowest speed Land at lowest altitude Exit at highest altitude Land/exit at right place Land in correct direction Dont fly over airports Keep planes separated Dont crash! Eight Simple Rules

31. ATST: Design Considerations

32. Measure KSAs Simulate wide range of job tasks Simulate a key job activity Possible Approaches

33. Maximize score variance No disadvantage for computer novices Short testing time Design Goals

34. Number of planes on the screen at one time Initial speed, level, heading, & location N of actions per plane Difficulty Factors: Planes

35. Speed of plane movement N and complexity of rules N of exits/airports N of controls N of control levels N of things to remember Difficulty Factors: Other

36. ATST: Scoring

37. Rule violations Separation errors Crashes Elapsed Time Record the Outcomes

38. N of violations of each rule N of separation errors N of crashes N of successful landings Duration of delays Types of Computed Scores

39. N of crashes & separation errors N of procedural errors Percent of successful flights Total delay time (handoff & en route) Á Priori Rational Scales

40. Moved flyovers based on data Standardize scores before combining and weighting Rescale scores if useful: reverse fix skewness make scores more sensitive to differences at higher levels Adjust Scales and Scores

41. ATST: Instructions & Practice

42. Will computer experience affect scores? If so, can you minimize its effect? If computer skill is job relevant, its not as serious. Computer Skill Effects

43. Clicking the mouse quickly Moving the mouse with speed and precision Using the keyboard Computer Skill Practice

44. Reduces effects of: learning ability computer game skills Simulation Practice Instructions with examples Practice a few things in mini-trial Give feedback and tips Several mini-trials

45. Add practice trials until more trials add little to: reliability validity Post-Instruction Practice

46. Pilot test with lots of trials Compute trial-total r Use last one or more trials with high rs Trial-criterion rs are even better if available Which Trials to Score

47. Score Weighting Methods

48. Final AT-SAT Validity u Validity, corrected for shrinkage, range restriction, and unreliability in criterion:.76 with composite criterion.78 with CBPM.38 with Ratings N 1029 1032 1053

49. Criteria u Reliability l.80 - CBPM (test-retest) l.71 - Ratings (interrater reliability) l.76 - Composite (mean of CBPM & Ratings) u Composite =.6 * CBPM +.4 * Ratings r CBPM J Ratings =.24

50. Uncorrected Test Validities u.41 Applied Math u.38 Analogies u.33 Angles u.33 Letter Factory u.32 Air Traffic Scenarios u.27 Time Wall u.25 Planes

51. Uncorrected Test Validities (contd) u.24 Dials u.23 Memory Retest u.19 Memory u.19 Scan u.17 Experiences Questionnaire u.14 Sound

52. Goals u Find best set of tests for the final battery u Determine scale weights u Measure validity: How well test battery predicts job performance

53. Decisions to be Made u Which predictors to keep u How to weight predictors

54. Decision 1: Choosing Predictors u Phase I- Evaluate predictors according to: l simple validity l incremental validity l fairness l group differences l test administration time u Predictors chosen by group consensus

55. Decision 1: Choosing Predictors u Phase 2: Optimal weighting algorithm l iterated regression l negative weights set to zero l maximizes R 2 while minimizing differences in group means, slopes, and intercepts u Many runs done while varying the relative importance of R 2 and group differences (10 parameters)

57. Validity vs. d -score Tradeoff

58. Decision 2: How to Weight Predictors u Alternatives Considered l regression weights l validity weights l optimal weights l unit weights u Chosen Weighting Method l mean of validity and optimal weights

59. Weighting Results: Validity WeightsValidity Shrunken Validity African- American d-score Regression.691.666-.85 Combined.682>.663-.81 Validity.664>.644-.88 Unit.604 -.92 Optimal.631.603-.55 Notes. N = 1029, Regression uses all 35 scales, others use the 26 scales in the final battery.

60. AT-SAT vs. OPM

61. Conclusions: Optimal Weights u An optimal weighting algorithm can balance the many considerations in selecting and weighting predictors u Optimal weighting tends to exclude too many predictors u Optimal weighting used in conjunction with another weighting method can perform very well

62. Conclusions: Validity Weights u In the current study, validity weights performed very well, especially when combined with optimal weights u Research is needed to compare the shrinkage (due to overfitting and predictor selection) and R 2 of validity weights to other weighting methods

63. Estimating Shrinkage in R u Shrinkage formula has shortcomings u Shrinkage formula corrects for overfitting but not predictor selection u Validity weighting - only selection shrinkage u Optimal weighting - shrinkage is severe because of extra parameters

64. Conclusions: Shrinkage u Research is needed to help estimate shrinkage of R (due to overfitting and predictor selection) using validity weights and optimal weights. u Research needed to compare validity weights, optimal weights, and regression weights in terms of shrunken R 2 (due to overfitting and predictor selection)

65. Setting the Cut Score

66. Regression Method u Set cut score on criterion u Compute corresponding score on predictor based on regression line u Works well when R-squared is high and good criterion is available

67. Criterion Cut Point Options u Cut score set at 5% (for example) of current incumbents distribution u Set cut score such that mean expected performance of candidates passing is at 60 th percentile (for example) of current incumbents u Set score at anchor on ratings scale (e.g., acceptable.

68. Cut Point Considerations u % of applicants passing u % of incumbents passing u % of each minority passing u adverse impact ratio u If low pass rate: l need many applicants, or l recruiting must target high-quality applicants

69. Questions?