1. Utility Analysis & Professional Qualifications in Test Administration BPS Level A and Level B (intermediate) courses Assessment in the Workplace

2. Overview w What is Utility Analysis (UA) ? w Traditional UA framework : SDy and costs of selection w Practical Example w Issues in UA w Alternative UA framework : Boudreau et al (1997) w Level A and Level B (Intermediate) w M21 Seen Paper

3. What is Utility Analysis ? a family of theories and measures designed to describe, predict and/or explain what determines the usefulness or desirability of decision options and to examine how information affects decisions (Boudreau, 1991) w Can be applied to selection processes, training, compensation, performance assessment, and internal staffing w Evolved to provide tools for better describing and communicating impact of HRM and Occ. Psy. interventions on organisational goals.

4. Traditional UA framework for utility analysis of new selection technique Top-down hiring Applicant Ability to Performance Value of PoolPredictIntervention New Intervention

5. Fundamental Processes in UA w The relationship between predictors and criteria (represented by r) w The nature of the criteria, represented by SDy w The nature of the selection process, represented by Z w The nature of the implementation process, represented by C.

6. The expected increase in output is directly proportional to the predictive validity of the test (Brogden, 1946) w Brogdens equation : Yearly saving per employee = (SDy x r v x Z) - [costs of selection] selected (£/$,etc) where : SDy = standard deviation of value in performance r v = predictive validity coefficient of the test Z = mean standard score of all applicants (selected and rejected) on the test Costs of Selection = administration costs per selected employee multiplied by the number of employees selected Hence, for the amount saved for N selected employees, staying with the company for T years : Total savings = [N x T x (SD y x r v x Z )] - (costs of selection)

7. How much more is a good worker worth than a poor worker (i.e. SDy) Schmitt and Hunter (1982) : 2 assumptions The value of a worker to a company is directly proportional to that workers performance (output) when the workers performance is measured on some criterion relevant to the company. If performance is normally distributed, then the value of performance should also be normally distributed.

8. When SDy is large vs when SDy is small If SDy is large (high) : then employees vary a lot in how much they are worth to the company... If SDy small (low) : then investment in selection is of less value, as there is less discrimination possible amongst workers ( >>> random selection ?!)

9. The costs of selection [ N.B. costs include costs of assessing those who are rejected as well as those that are selected ] Costs must include all costs (i.e. labour time, materials, selection programme development costs, etc.) If the cost of assessing each applicant is C and SR (selection ratio) is the proportion of applicants selected : Costs per selected applicant = C / SR. Costs of selection = (costs per selected applicant) x (number of applicants selected)

10. Examples of Utility Analysis Results Schmitt & Hunters estimates of predicted savings due to large scale implementation of improved selection methods : w Use of Computer Aptitude Battery by US Federal Government : $64,000 per selectee, over 9 years. w In US Federal Government selection, replacement of ability tests and work sample tests by less valid predictors would lead to losses of $3.12 billion (job tryout) to $15.8 billion (age) per year. w General improvements in the Philadelphia Police Force (5000 employees) : $18 million/year w General improvements in selection - US National Economy $87-$152 billion/year.

11. Practical Exercise Old Selection Test SDy = $30,000, r v = 0.2, Z = 1.0 Costs per selected employee = £1,000 10 people (N = 10) selected who stay on average 4 years (T = 4) New Selection Test SDy = $30,000, r v = 0.4, Z = 1.0, N = 10, T = 4 Costs per selected employee = £2,000 w Calculate the total savings (larger figure = more saved) for the old test and the new test. w Should the old or the new test be used ?

12. Special Issues w To what extent can the validity reported for a test in a manual be applied to your selection context (Validity Generalisation & synthetic validity) w Real life issues not considered by traditional utility analysis Inflation (selection costs need to be adjusted) Variable costs of employees Taxation (of additional profits) Lost investment elsewhere Number of good candidates available? Do not overlook the old selection system (typically assume validity is zero). The real worth of improved selection

13. Boudreau et al (1997) Actual Hiring Method ApplicantAbility to PoolPredict currentaspects of VALUE interventionsnewperformance OF intervention(s) INTERVENTION

14. Boudreau et al (1997) w Multivariate validity over time (the Black Box of validity) temporal stability of validity multidimensional criteria (the nature of SDy) w Using employee movement to represent multiple criteria w The interaction between multiple predictors and multiple criteria w The Black Box of Z w Is maximising organisational performance (££££s) always an organisations main objective ?

15. Professional Qualifications in Test Administration BPS Certificates of Competence in Occupational Testing w BPS website - for level A and level B (int.) General Information Packs (http://www.bps.org.uk) w Is it worth me taking these additional courses? w Course dates and costs - see MSc noticeboards w Course content & structure Level A : Ability tests, aptitude tests, interest inventories Level B (Intermediate) : Hogans Personality Inventory (HPI) Both 3 days (9am-5pm), with pre- & post-course work