1. What’s New in the I/O Testing and Assessment Literature That’s Important for Practitioners?
Paul R. Sackett

2. New Developments in the Assessment of Personality

3. Topic 1: A faking-resistant approach to personality measurement
Tailored Adaptive Personality Assessment System (TAPAS)
Developed for Army Research Institute by Drasgow Consulting Group
Multidimensional Pairwise Preference Format combined with applicable Item Response Theory model
Items are created by pairing statements from different dimensions that are similar in desirability and trait “location”
Example item: “Which is more like you?”
__1a) People come to me when they want fresh ideas.
__1b) Most people would say that I’m a “good listener”.

4. A faking-resistant approach to personality measurement (continued)
Extensive work show it’s faking-resistant
Non-operational field study in Army show useful prediction of attrition, disciplinary incidents, completion of basic training, adjustment to Army life, among other criteria
Now in operational use on a trial basis
Drasgow, F., Stark, S., Chernyshenko, O. S., Nye, C. D., and Hulin, C. L. (2012). Development of the Tailored Adaptive Personality Assessment System (TAPAS) to Support Army Selection and Classification Decisions. Technical Report 1311, Army Research Institute

5. Topic 2: The Value of Contextualized Personality Items
A new meta-analysis documents the higher predictive power obtained by “contextualizing” items (e.g., asking about behavior at work, rather than behavior in general)
Mean r with supervisory ratings for work context vs. general:
Conscientiousness: .30 vs .22
Emotional Stability: .17 vs. 12
Extraversion: .25 vs. .08
Agreeableness: .24 vs. .10
Openness: .19 vs. .02
Shaffer, J.A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65,

6. Topic 3: Moving from the Big 5 to Narrower Dimensions
DeYoung, Quilty and Peterson (2007) suggested the following:
Neuroticism:
Volatility - irritability, anger, and difficulty controlling emotional impulses
Withdrawal - susceptibility to anxiety, worry, depression, and sadness
Agreeableness:
Compassion - empathetic emotional affiliation
Politeness - consideration and respect for others’ needs and desires
Conscientiousness:
Industriousness - working hard and avoiding distraction
Orderliness - organization and methodicalness
Extraversion:
Enthusiasm - positive emotion and sociability
Assertiveness - drive and dominance
Openness to Experience:
Intellect - ingenuity, quickness, and intellectual engagement
Openness - imagination, fantasy, and artistic and aesthetic interests
DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007). Between facets and domains: 10 Aspects of the Big Five, Journal of Personality and Social Psychology, 93,

7. Moving from the Big 5 to Narrower Dimensions (continued)
Dudley et al (2006) show the value of this perspective
Four conscientiousness facets: achievement, dependability, order, and cautiousness
Validity was driven largely by the achievement and/or dependability facets, with relatively little contribution from cautiousness and order
Achievement receives the dominant weight in predicting task performance, while dependability receives the dominant weight in predicting counterproductive work behavior
Dudley NM, Orvis KA, Lebiecki JE, Cortina JM A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. J. Appl. Psychol. 91:40-57

8. Topic 4: The Use of Faking Warnings
Landers et al (2011) administered a warning after 1/3 of the items to managerial candidates exhibiting what they called “blatant extreme responding”.
Rate of extreme responding was halved after the warning
Landers, R. N., Sackett, P. R., & Tuzinski, K. A. (2011). Retesting after initial failure, coaching rumors, and warnings against faking in online personality measures for selection. Journal of Applied Psychology, 96(1), 202.

9. More on the Use of Faking Warnings
Nathan Kuncel suggests three potentially relevant goals when individuals take a personality test:
- be impressive
- be credible
- be true to oneself

10. More on the Use of Faking Warnings
Jenson and Sackett (2013) suggested that “priming” concern for being credible could reduce faking.
Test-takers who scheduled a follow-up interview just before taking the personality test obtained lower scores than those who did not
Jenson, C. E., and Sackett, P. R. (2013). Examining ability to fake and test-taker goals in personality assessments. SIOP presentation.

11. New Developments in the Assessment of Cognitive Ability

12. A cognitive test with reduced adverse impact
In 2011, SIOP awarded its M.Scott Myers Award for applied research to Yusko, Goldstein, Scherbaum, and Hanges for the development of the Siena Reasoning Test
This is a nonverbal reasoning test, using unfamilar item content, such as made-up words (if a GATH is larger than a SHET…) and figures
Concept is that adverse impact will be reduced by eliminating content with which groups have differential familiarity

13. Validity and subgroup d for Siena Test
Black-White d commonly in the range
Sizable number of validity studies, with validities in the range commonly seen for cognitive tests.
In one independent study, HumRRO researchers included Siena along with another cognitive test; corrected validity .45 for other test (d = 1.); .35 for Siena (d = .38) (SIOP 2010: Paullin, Putka, and Tsacoumis)

14. Why the reduced d?
Somewhat of a puzzle. There is a history of using non-verbal reasoning tests
Raven’s Progressive Matrices
Large sample military studies in Project A
But these do not show the reduced d that is seen with the Siena Test
Things to look into: does d vary with item difficulty, and how does Siena compare with other tests?
(Note: Nothing published to date that I am aware of. Some powerpoint decks from SIOP presentations can be found online: search for “Siena Reasoning Test”)

15. New Developments in Situational Judgment Testing

16. Sample SJT item
You find yourself in an argument with several co-workers about who should do a very disagreeable, but routine task. Which of the following would be the most effective way to resolve this situation?
(a) Have your supervisor decide, because this would avoid any personal bias.
(b) Arrange for a rotating schedule so everyone shares the chore.
(c) Let the workers who show up earliest choose on a first-come, first-served basis.
(d) Randomly assign a person to do the task and don't change it.

17. Key findings Extensive validity evidence
Can measure different constructs (problem solving, communication skills, integrity,etc.)
Incremental validity over ability and personality
Small subgroup differences, except for cognitively-oriented SJTs
Items can be presented in written form or by video; recent move to animation rather than recording live actors

18. Lievens, Sackett, and Buyse, T. (2009) comparing response instructions
Ongoing debate re “would do” vs. “should do” instructions
Lievens et al. randomly assigned Belgian medical school applicants to “would do” or “should do” in operational interpersonal skills SJT; did the same with a student sample

19. Lievens, Sackett, and Buyse, T. (2009) comparing response instructions
In operational setting, all gave “should do” responses
So: we’d like to know “would do”, but in effect, can only get “should do”

20. Arthur et al (2014): comparing response formats
Compared 3 options:
Rate effectiveness of each response
Rank the responses
Choose best and worst response
20-item integrity-oriented SJT
Administered to over 30,000 retail/hospitality job applicants
On-line admin; each format used for one week

21. “Rate each response” emerges as superior
Higher reliability
Lower correlation with cognitive ability
Smaller gender mean difference
Higher correlation with conceptually relevant personality dimensions (conscientiousness, agreeableness, emotional stability)
Follow-up study with student sample
Higher retest reliabilty
More favorable reactions

22. Question: how “situational” is situational judgment?
Krumm et al. (in press)
Question: how “situational” is situational judgment?
Some suggest SJTs really just measure general knowledge about appropriate social behavior
So Krumm et al. conducted a clever experiment: they “decapitated” SJT items
Removed the stem – just presented the responses

23. 559 airline pilots completed 10 items each from
Airline pilot knowledge SJT
Integrity SJT
Teamwork SJT
Overall, mean scores are 1 SD higher with the stem
But for more than half the items, there is no difference with and without stem
So stem matters overall, but is irrelevant for lots of SJT items
Depends on specificity of stem content

24. “You are flying an “angel flight” with a nurse and noncritical child patient, to meet an ambulance at a downtown regional airport. You filed visual flight rule: it is 11:00 p.m. on a clear night, when, at 60 nm out, you notice the ammeter indicating a battery discharge and correctly deduce the alternator has failed. Your best guess is that you have from 15 to 30 min of battery power remaining. You decide to:
(a) Declare an emergency, turn off all electrical systems, except for 1 NAVCOM and transponder, and continue to the regional airport as planned.
(b) Declare an emergency and divert to the Planter’s County Airport, which is clearly visible at 2 o’clock, at 7 nm.
(c) Declare an emergency, turn off all electrical systems, except for 1 NAVCOM, instrument panel lights, intercom, and transponder, and divert to the Southside Business Airport, which is 40 nm straight ahead.
(d) Declare an emergency, turn off all electrical systems, except for 1 NAVCOM, instrument panel lights, intercom, and transponder, and divert to Draper Air Force Base, which is at 10 o’clock, at 32 nm.”

25. Arthur, W. , Jr. , Glaze, R. M. , Jarrett, S. M. , White, C. D
Arthur, W., Jr., Glaze, R. M., Jarrett, S. M., White, C. D., Schurig, I., & Taylor, J. E. (2014). Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion. Journal of Applied Psychology, 99(3),
Krumm, S, Lievens, F., Huffmeier,J., Lipnevich, A., Bendels,H., and Hertel, G.(in press). How “situational” is judgment in situational judgment tests? Journal of Applied Psychology.
Lievens, F., Sackett, P. R, and Buyse, T. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94,

26. New Developments in Integrity Testing

27. Two meta-analyses with differing findings
Ones, Viswesvaran, and Schmidt (1993) is the “classic” analysis of integrity test validity.
found 662 studies, including many where only raw data was provided (i.e., no write-up). Info sharing from many publishers
In 2012, Van Iddekinge et al conducted an updated meta-analysis
applied strict inclusion rules as to what studies to include (e.g., reporting of study detail)
104 studies (including 132 samples) met inclusion criteria.
30 publishers contacted; only 2 shared info.
Both based bottom line conclusions on studies using a predictive design and a non-self report criterion.

28. Predicting Counterproductive Behavior
K N Mean Validity
Ones et al – overt tests
Ones et al- personality
based tests
Van Iddekinge et al

29. Why the difference?
Not clear. A number of factors do not seem to be the cause:
Differences in types of studies examined (e.g., both excluded studies with polygraph as criteria)
Differences in corrections (e.g., unreliability)
Several factors may contribute, though this is speculation
Some counterproductive behaviors may be more predictable than others, but all are lumped together in these analyses
Given reliance in both on studies not readily available to public scrutiny, this won’t be resolved until further work is done

30. Broader questions
This raises broader issues about data openness policies
Publisher obligations?
Researcher obligations?
Journal publication standards?
Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679 –703
Van Iddekinge, C. H., Roth, P. L., Raymark, P. H., & Odle-Dusseau, H. N. (2012). The criterion-related validity of integrity tests: An updated meta-analysis. Journal of Applied Psychology, 97, 499 –530.

31. New Developments in Using Vocational Interest Measures

32. BUT: how many studies in this meta-analysis?
Since Hunter and Hunter (1984), interest in using interest measures for selection has diminished greatly
They report a meta-analytic estimate of validity for predicting performance as .10
BUT: how many studies in this meta-analysis?
3!!!

33. New meta-analysis by Van Iddekinge et al. (2011)
Lots of studies (80)
Mean validity for a single interest dimension: .11
Mean validity for a single interest dimension relevant to the job in question: .23
Other studies suggest incremental validity over ability and personality

34. The “catch”: studies use data collected for research purposes
Concern that candidates can “fake” a job-relevant interest profile
I expect interest to turn to developing faking-resistant interest measures

35. Van Iddekinge, C. H. , Roth, P. L. , Putka, D. J. , & Lanivich, S. E
Van Iddekinge, C. H., Roth, P. L., Putka, D. J., & Lanivich, S. E. (2011). Are you interested? A meta-analysis of relations between vocational interests and employee performance and turnover. Journal of Applied Psychology, 96(6), 1167.
Nye, C. D., Su, R., Rounds, J., & Drasgow, F. (2012). Vocational interests and performance a quantitative summary of over 60 years of research. Perspectives on Psychological Science, 7(4),

36. New Developments in Using Social Media

37. Van Iddekinge et al (in press)
Students about to graduate made Facebook info available
Recruiters rated profile on 10 dimensions
Supervisors rated performance a year later
Facebook ratings did not predict performance
Higher ratings for women than men
Lower ratings for Blacks and Hispanics than Whites
Van Iddekinge, C. H., Lanivich, S. E., Roth, P. L., & Junco, E. (in press). Social Media for Selection? Validity and Adverse Impact Potential of a Facebook-Based Assessment. Journal of Management.

38. Distribution of Performance

39. Is performance normally distributed?
We’ve implicitly assumed this for years
Data analysis strategies assume normality
Evaluations of selection system utility assume normality
O’Boyle and Aguinis (2012) offer hundreds of data sets, all consistently showing that a “power law” distribution fits better
This is a distribution with the largest number of observations at the very bottom, with the number of observations then dropping rapidly

41. The O’Boyle and Aguinis data
They argue against looking at ratings data, as ratings may “forced” to fit a normal distribution
Thus they focus on objective data
Tallies of publication in journals
Sports performance (e.g., golf tournaments won, points scored in NBA)
Awards in arts and letters (e.g. Number of Academy Award nominations)
Political elections (number of terms to which one has been elected)

42. An alternate view
“Job performance is defined as the total expected value of the discrete behavioral episodes an individual carries out over a standard period of time” (Motowidlo and Kell, 2013)

43. Aggregating individual behaviors affects distribution

44. Including all performers affects distribution

45. Equalizing opportunity to perform

46. References
O’Boyle Jr. E., & Aguinis, H. (2012). The best and the rest: Revisiting the norm of normality of individual performance. Personnel Psychology, 65(1), 79.
Beck, J., Beatty, A. S., and Sackett, P. R. (2014) On the distribution of performance: A reply to O’Boyle and Aguinis. Personnel Psychology, 67,