1. The variability of assessment centre validities: Subject to purpose? Kim Dowdeswell, Senior Research Consultant & Industrial Psychologist 30 th Assessment Centre Study Group Conference, 18 th March 2010 ©2010 SHL Group Limited

2. 2 Presentation Outline Signposting the discussion Introduction Assessment centre validation research over the years Assessment vs. development centre usage trends Comparing validities for overall assessment ratings vs. dimension ratings Potential differences in validities between assessment centre purposes and approach Questions?

3. 3 ©2010 SHL Group Limited Introduction On assessment centres Assessment centre methods incorporate three features: –The use of multiple assessment techniques –Standardised methods of making inferences from such techniques –Pooled judgements of multiple assessors in rating each candidate’s behaviour For three major purposes (International Task Force on Assessment Center Guidelines, 2009): –To predict future behaviour for decision-making –To diagnose development needs –To develop candidates on dimensions of interest

4. AC validation research over the years ©2010 SHL Group Limited An overview of meta-analytic research evidence

5. 5 ©2010 SHL Group Limited AC Validation Research The ‘gold standard’ Gaugler et al. (1987): Corrected mean validity coefficient of 0.37 –Meta-analysis of 50 assessment centre studies –Relation between overall assessment ratings and various criteria

6. 6 +1PERFECT PREDICTION.63.60.54.51.40.37.35.26.18.02 0 -0.01 Ability and Structured Interview Ability and Work sample Work Sample Tests Structured Interviews Ability Tests Personality Tests Assessment Centres Bio data References Years Job Experience Graphology RANDOM PREDICTION Age The Validity Ladder Schmidt & Hunter (1998) Criterion: Overall Job Performance

7. 7 ©2010 SHL Group Limited AC Validation Research The ‘gold standard’ Gaugler et al. (1987): Corrected mean validity coefficient of 0.37 –Meta-analysis of 50 assessment centre studies –Relation between overall assessment ratings and various criteria Differences observed in validity coefficients i.t.o.: By Criteria:By AC Purpose: Job performance0.36Promotion0.30 Potential ratings0.53Early identification0.46 Dimension ratings0.33Selection0.41 Training performance0.35Research0.48 Career advancement0.36

8. 8 ©2010 SHL Group Limited AC Validation Research Over the years Validity coefficients of assessment centres seem to be dropping: 1987 -> 2007 shows a statistically significant drop: StudyYearValidity coefficient Gaugler et al.19870.37 Hermelin et al.20070.28 StudyYear95% CI band Gaugler et al.19870.30 ≤ ρ ≤ 0.42 Hermelin et al.20070.24 ≤ ρ ≤ 0.32

9. 9 ©2010 SHL Group Limited AC Validation Research Challenges to conducting AC validation Sampling error –AC time and cost = small samples Moderate to severe levels of range restriction –Starting with small samples -> even fewer appointments Reliability of supervisor ratings of performance / potential –A notoriously common problem in validation research

10. 10 ©2010 SHL Group Limited AC Validation Research Challenges to conducting AC validation Sampling error –AC time and cost = small samples Moderate to severe levels of range restriction –Starting with small samples -> even fewer appointments »Hermelin et al. (2007) put indirect range restriction forward as an explanation for the lower results observed; with cost considerations, ‘modern’ AC participants subject to more pre-selection than previously Reliability of supervisor ratings of performance / potential –A notoriously common problem in validation research

11. Assessment vs. development centre usage trends ©2010 SHL Group Limited The influence of purpose

12. 12 ©2010 SHL Group Limited AC vs. DC Usage Trends The influence of purpose The use of ACs for development has been increasing over the years Popular purposes for utilising ACs in surveyed US organisations (Spychalski et al., 1997): –Selection (50.0%) / Promotion (45.8%) –Development planning (39.2%) Popular purposes for utilising ACs in surveyed SA organisations (Krause et al., 2010): –Selection & Development (65%) –Selection alone (22%) –Development alone (13%)

13. 13 ©2010 SHL Group Limited AC vs. DC Usage Trends The influence of purpose: Selection ACs Selection ACs are designed to identify the best candidate for a job Features in ‘Selection Centres’ (Spychalski et al., 1997): –Assessors serve many times annually; ample opportunity to keep skills current –Assessors are almost always asked to compile overall performance ratings for selection purposes –‘Selection Centre’ data validated more frequently than in centres used for other purposes More frequent likelihood of validation to be prepared for possible legal challenges to selection decisions based on AC data?

14. 14 ©2010 SHL Group Limited AC vs. DC Usage Trends The influence of purpose: Development ACs Goals of developmental assessment centres vary: –Identifying training needs –Formulating personalised developmental needs & action plans –Developing skills on the basis of immediate feedback and practice Features in ‘Development Planning Centres’ (Spychalski et al., 1997): –Fewer candidate selection mechanisms used, with heavy reliance on supervisor data –Assessors conduct lengthy discussion sessions, with other assessors and with candidates in feedback sessions –‘Development Planning Centres’ infrequently validated

15. 15 ©2010 SHL Group Limited AC vs. DC Usage Trends The influence of purpose: Assessor evaluations Possible implications of AC focus shifting from selection to development? “Assessors may evaluate candidates differently, depending on whether their ratings will serve a selection purpose (i.e., ‘yes/no’ decision) or a developmental purpose (i.e., identification of strengths and weaknesses).” (Lievens & Klimoski, 2001) The authors noted (in 2001) that there was no assessment centre research manipulating such variables that they knew of

16. Comparing validities for overall assessment ratings vs. dimension ratings ©2010 SHL Group Limited Which yield higher validities?

17. 17 ©2010 SHL Group Limited Comparing Validities Overall assessment vs. dimension ratings ‘Selection Centres’ typically use an overall assessment rating (OAR) to inform selection / promotion decisions –OAR-based validity evidence e.g. Gaugler et al. (1987); Hermelin et al. (2007) ‘Development Centres’ typically use dimension ratings to facilitate detailed feedback with participants about their strengths and weaknesses –Dimension-based validity evidence e.g. Arthur et al. (2003)

18. 18 ©2010 SHL Group Limited Comparing Validities Overall assessment vs. dimension ratings Arthur et al. (2003) Criterion-related validity of AC dimensions compared to OARs Dimension Validity Coefficient OAR Validity Coefficient Problem Solving0.39 0.37 (Gaugler et al., 1987) Influencing Others0.38 Organizing & Planning0.37 Communication0.33 Drive0.31 Consideration/awareness of others 0.25

19. 19 ©2010 SHL Group Limited Comparing Validities Overall assessment vs. dimension ratings Arthur et al. (2003) Criterion-related validity of regression-based composite of AC dimensions compared to OARs DimensionR OAR Validity Coefficient Problem Solving0.39 0.37 (Gaugler et al., 1987) Influencing Others0.43 Organizing & Planning0.44 Communication0.45 Drive0.45 Consideration/awareness of others 0.45

20. 20 ©2010 SHL Group Limited Comparing Validities Overall assessment vs. dimension ratings Arthur et al. (2003) The use of OARs may result in an underestimate of criterion- related validity of assessment ratings Predictive validity of AC composite scores derived from dimension weights can be enhanced if dimension intercorrelations reduced May not need as many dimensions as have typically been used AC dimensions -> single composite score in same manner as typically used to combine scores from multipredictor test batteries

21. Potential differences in validities between AC purposes & approach ©2010 SHL Group Limited What can we learn about increasing AC validity?

22. 22 ©2010 SHL Group Limited Differences in Approach & Purpose What can we learn about increasing validity? We have seen validity findings are influenced by: –The criteria used »E.g. Dimension ratings (0.33) vs. Potential ratings (0.53) –The purpose of the AC »E.g. Promotion (0.30) vs. Early Identification of Potential (0.46) –The approach used in compiling candidate scores »E.g. OARs (0.37) vs. individual dimensions (0.25-0.39) vs. regression- based composite of dimensions (0.45) Good research design & methodology is key

23. 23 ©2010 SHL Group Limited Differences in Approach & Purpose What can we learn about increasing validity? Literature fairly consistent i.t.o. other characteristics of ‘more highly valid’ ACs: –Using psychologists as assessors rather than managers »Concern: Spychalski et al (1997) found that only 5.7% of cases utilised psychologists as assessors –Limit the number of dimensions evaluated »Concern: Krause et al. (2010) found only 20% of SA organisations evaluated 5 dimensions or less in ACs –Evaluate the ‘right’ dimensions; critical for success as identified by role analysis »Further: Arthur et al. (2003) found some dimensions are more valid than others

24. 24 ©2010 SHL Group Limited When considering several studies reporting on developmental assessment centres, Lievens & Klimoski (2001) noted as a limitation the majority of studies did not relate DC ratings to external criteria “However, in assessment centers conducted for developmental purposes other constructs might serve as more relevant criteria” “When validating or otherwise evaluating DACs, the appropriate criterion is change in participants’ understanding, behavior, and proficiency on targeted dimensions” (International Task Force on Assessment Center Guidelines, 2009) Differences in Approach & Purpose A final point on ACs vs. DCs & validities…

25. 25 ©2010 SHL Group Limited We should not lose sight of the purpose of the Assessment Centre (Howard, 2009): –Selection/Promotion: »Help find the right person for the job –Diagnosis/Development: »Better choice of development activities? –Succession/Placement: »Help find the right job for the person? Differences in Approach & Purpose A final point on ACs vs. DCs & validities…

26. 26 Questions…

27. 27 ©2010 SHL Group Limited References Arthur Jr, W., Day, E.A., McNelly, T.L. & Edens, P.S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125-154. Gaugler, B.B., Rosenthal, D.B., Thornton, G.C. & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493-511. Hermelin, E., Lievens, F. & Robertson, I.T. (2007). The validity of assessment centres for the prediction of supervisory performance ratings: A meta-analysis. International Journal of Selection & Assessment, 15(4), 405-411. Howard, A. (2009). Making assessment centers work they way they are supposed to. Keynote address at the 29 th Assessment Centre Study Group Conference, Stellenbosch, South Africa, March 2009. International Task Force on Assessment Center Guidelines. (2009). Guidelines and ethical considerations for assessment center operations. International Journal of Selection & Assessment, 17(3), 243-253).

28. 28 ©2010 SHL Group Limited References Krause et al. (2010). State of the art assessment centre practices in South Africa: survey results, challenges, and syggestions for improvement. Keynote address at the 30 th Assessment Centre Study Group Conference, Stellenbosch, South Africa, 18-19 March 2010. Lievens, F. & Conway, J.M. (2001). Dimension and exercise variance in assessment centre scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86(6), 1202-1222. Lievens, F. & Klimoski, R.J. (2001). Understanding the assessment center process: Where are we now? In C.L. Cooper & I.T. Robertson (Eds.) International Review of Industrial and Organizational Psychology vol. 16 (pp. 245-286). Chicester: John Wiley & Sons, Ltd. Spychalski, A.C., Quinones, M.A., Gaugler, B.B. & Pohley, K. (1997). A survey of assessment center practices in organizations in the United States. Personnel Psychology, 50, 71-90.