SELECTION & ASSESSMENT SESSION THREE: MEASURING THE EFFECTIVENESS OF SELECTION METHODS

IN THIS SESSION A review of the various types of validity we would examine in a selection method The importance of validity generalisation analysis A review of the various types of reliability we would examine in a selection method An analysis of a personality measure in terms of its validity and reliability

VALIDITY IN SELECTION The ability of a selection method to measure what it purports to measure Face validity Content validity Construct validity Criterion-related validity

FACE & CONTENT VALIDITY The selection method taken at face value Importance to candidates A dangerous short cut The coverage of a conceptual domain Content under-representation and construct- irrelevant variance

CONSTRUCT VALIDITY Measuring the thing we’re supposed to measure Critical importance of good definitions The need for convergent and divergent comparisons What you will find in a test manual

CRITERION-RELATED VALIDITY The relationship between the selection method and some external criterion Concurrent Validity – Relationship between results and another assessment at the same time Predictive Validity – Looks to the future and assesses people at two different points

HOW VALID ARE THESE METHODS? Application forms0.12 References0.26 Interviews (all types)0.35 Structured interviews0.51 Cognitive ability tests0.51 Personality measures0.40 Biodata0.35 Assessment centres0.46 (Robertson & Smith 2001)

REASONS FOR LOW VALIDITY Moderator variables Situational specificity The 0.3 barrier Validity Generalisation Analysis

VALIDITY GENERALISATION ANALYSIS A typical validity study has four limitations: when you correct for these, validity coefficient increases: – Sampling error – Restricted range – Criterion reliability – Test reliability

APPLICATIONS OF VGA Incremental validity Move away from supervisor ratings Validity moderated by job complexity Job families

IMPLICATIONS OF VGA Raises true validity of selection Allows export of ability tests The end of job analysis? Criticisms of reporting bias, restricted range and criterion reliability

PRACTICAL EXERCISE Look at the attached extract from the BPS Test Review of the Giotto Integrity Measure. In small groups discuss the problems of validity and how you would advise a client to use the measure for selection purposes

RELIABILITY IN SELECTION How much weight or importance can we put on the assessment score? Should expect correlations of 0.9 and reject anything less than 0.7 Looks to minimise unsystematic errors Stability, consistency and equivalence

STABILITY OF ASSESSMENT Repeatability over time Assumption that things don’t change Beware carry-over effect and practice effect

CONSISTENCY OF ASSESSMENT Looks at internal reliability of the measure Uses two main methods: – Split-Half Method – Cronbach Alpha Coefficient

EQUIVALENCE OF RESULTS Also know as Parallel Forms Reliability Measures equivalence between different versions of the same test Better than test-retest method as avoids threats from maturity, practice etc

PRACTICAL EXERCISE Look at the attached extract from the BPS Test Review of the Giotto Integrity Measure. In small groups discuss the problems of reliability and how you would advise a client to use the measure for selection purposes

SELECTION & ASSESSMENT SESSION FOUR: DESIGNING BETTER ASSESSMENT PROCEDURES

IN THIS SESSION For the rest of today’s session we will return to the recruitment advert for a Senior Business Psychologist we first encountered in Session One of this module. Have a look again at the advert on the next slide

PRACTICAL TASK Your task is to advise OPS how they might properly and rigorously assess the competences they are interested in Remembering what we’ve said about validity and reliability today, your task is firstly to properly define one of the constructs in the advert then think of a valid and reliable way to measure it The competences in question are: – strong interpersonal skills, strong influencing skills, creativity, effective planning, effective organisational skills, an eye for quality