Finding the ‘Right’ Students: The Search for Predictive Validity in Applicant Screening for the Health Sciences Rosemary Lysaght, Ph.D. Catherine Donnelly, M.Sc. Michelle Villeneuve, M.Sc. School of Rehabilitation Therapy

Background Impetus for examination of our admissions process was the creation of the new masters entry-level program in 2004 Goals for admission included: An evidence-based admissions process Increase the applicant pool Efficiency in admission process

Overall Aim of Selection To select students who will: Succeed in the academic program Perform credibly in professional practice Possess the traits of character and ethical values desired of a professional person (Nayer, 1992)

Commonly Used Selection Criteria in Health Sciences Pre-admission academic grades Discipline-specific aptitude tests (MCAT, HOAE) Interviews Written submissions Letters of reference Prerequisites (Auriemma, 2002; Salvatori, 2001)

Predictors of academic success Evidence across disciplines supports the predictive validity of –Pre-admission academic grades No clear support found for –Discipline-specific aptitude tests –Interviews –Written submissions –Letters of reference –Prerequisites (Bridle, 1987; Caplan et al, 1996; Howard & Jerosh-Herold, 2000; Lewis & Smith, 2002; Kirchner et al., 2000; Kirchner & Holm, 1997; Salvatori, 2001)

Predictors of clinical performance Unclear relationship between pre-admission performance and clinical performance Valid outcome measures difficult to identify Lack of consistent raters Variability across settings (Howard & Jerosch-Herold, 2000; Kirchner & Holm, 1997; Katz & Mosey, 1980; Tan et al., 2004)

Research Questions 1.What admissions screening tools best predict academic performance in a masters level OT program?  Selected academic courses and  Clinical performance 2.Does undergraduate coursework predict success in topically-related coursework?

Method Analysis of existing data for 128 students admitted to the OT masters program –Multiple regression –Models created for each research question Project received approval by the Queen’s University’s Research Ethics Board Sample included 3 cohorts of students (1 st three years of the new professional masters program)

Factors Considered in OT Admissions –GPA –Academic transcript –Letter of intent –2 referee rating forms/letters –Supporting data (foreign/non-traditional applicants)

Predictor Variables Undergraduate Grade Point Average (GPA) – from ORPAS Letter of Intent (LOI) Average rating, 2 raters Referee Rating 5 point scale, 12 items Average total score of 2 external referees

Predictor Variables (con’t) Additional ratings were created for each student based on file review (1 = weak to 5 = exceptional)  Physical Sciences Experience/Preparation e.g. Anatomy, physiology, kinesiology (r for 2 raters =.98)  Social Sciences Experience/Preparation e.g. Psychology, Sociology, Family Studies (r=.91)  Experience with People with Disabilities/Vulnerable Populations Rated for number, duration, and relevance (r=.79) Information drawn from Letter of Intent, referee letters and experience questionnaire

Criterion Variables Program GPA (1 st year) Communication Skills Practicum Targeted course grades in: Physical Determinants of Occupation Cognitive-Neurological Determinants Psycho-Emotional Determinants

Descriptive Statistics - Predictor Variables VariableNMeanSD Undergraduate GPA Letter of Intent Rating Referee Rating Bkg. Physical Sciences Bkg. Social Sciences Bkg. Experience with PWD

Descriptive Statistics - Criterion Variables VariableNMeanSD First Year GPA Communication Skills Rating Physical Determinants Grade Cogntive-Neuro Det. Grade Psycho-Emotional Det. Grade

Results – First Year GPA Predictor Variables: GPA, Letter of Intent, Referee Rating Model is significant (R 2 =.101; F= 4.65, p <.001) GPA is only variable with significant beta score (p =.005) Significant correlations between all three variables and program GPA

Results – Communication Skills Predictor Variables: GPA, Letter of Intent, Referee Rating, all experience ratings, Model is not significant (R 2 =.021; F=.62) Referee rating is significantly correlated with performance rating No background experience ratings were correlated with performance rating

Results – Physical Determinants Predictor Variables: GPA, all experience ratings Result: Model not significant (R 2 =.009; F=.275) No significant correlations between any background experience rating and outcome

Results – Cognitive-Neuro Predictor Variables: GPA, all experience ratings Result: Model is significant (R 2 =.14; F=5.0, p =.004) GPA only variable with significant beta score (p =.00) Background in physical sciences was significantly negatively correlated with outcome

Results – PsychoEmotional Det. Predictor Variables: GPA, all experience ratings Result: Model is significant (R 2 =.082; F = 2.6, p<.05) Social science bkgd only variable with significant beta score (-.2 ) GPA is significantly positively correlated with outcome, while SS bkgd is significantly negatively correlated.

Additional Observations The contribution of the undergraduate GPA to all models was reduced by the addition of the 3 rd cohort, which had significantly higher GPA ratings than the first 2 cohorts and less spread in scores Referee ratings also had small but significant correlations with –physical determinants grade (r =.18, p =.02) –communication skills grade (r =.25, p =.003)

Conclusions Findings relative to GPA as a significant predictor of academic performance in a health science program supports previous research Other positive correlates with 1 st year GPA: –Referee ratings –LOI ratings suggest that these measures have some value in the admissions process. Referee ratings have even broader potential value, given positive correlations with performance in Physical Determinants & Communications Skills courses.

Conclusions (con’t) No support for the requirement of specific academic pre-requisites Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance

No support for the requirement of specific academic pre-requisites Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance

Discussion Fairness of admissions process –Pre-requisite requirements not justified if predictive validity of courses not substantiated –No control over authorship of LOI Inherent biases in process –LOI requirement may bias selection towards persons from the same culture who highlight issues salient to that culture, females, and strong writers –Applicants with strong social science background may be over- represented if GPA is primary selection factor Impact on program –Elimination of pre-requisites broadens applicant pool, may result in higher calibre class –More diversity of students in program/field

Discussion Role of Pre-Requisites and other Screening Tools –LOI, Referee ratings useful as screen out, rather than selection criteria? –Certain pre-requisites may make academic progress easier for student –Admissions requirements may have value beyond identification of best applicants Credibility of process and applicant Interviews may help form relationships, promote program

Limitations Findings limited to one health science program format within a Canadian context, and may not generalize Other potential screening tools not available for consideration in this study Course grades are subject to unsystematic scoring errors Ratings of background experience and LOI subject to rater error

References Auriemma, D. (2002). Admission methods of professional occupational therapy programs in the united states. Education Special Interest Section Quarterly, 12(3), 1-4. Bridle, M. J. (1987). Student selection: A comparison of three methods... queen's university occupational therapy program. Canadian Journal of Occupational Therapy, 54(3), Caplan, R.M, Kreiter, C., & Albanese, M. (1996). Preclinical science course ‘preludes’ taken by premedical students: do they provide a competitive advantage? AMJ, Howard, L., & JeroschHerold, C. (2000). Can entry qualifications be used to predict fieldwork and academic outcomes in occupational therapy and physiotherapy students? British Journal of Occupational Therapy, 63(7), Katz, G.M., & Mosey, A.C. (1980). Fieldwork performance, academic grades, and pre-selection criteria of occupational therapy students. American Journal of Occupational Therapy, 34(12), 794 – 800.

Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and clinical performance of occupational therapy students in an entry- level master's program. American Journal of Occupational Therapy, 51(9), Kirchner, G. L., Stone, R. G., & Holm, M. B. (2000). Use of admission criteria to predict performance of students in an entry-level master's program on fieldwork placements and in academic courses. Occupational Therapy in Health Care, 13(1), Lewis, M., Smith, S. (2002). Selection of pre-registration physiotherapy students: Changing to a more objective process. Physiotherapy, 88(11), 688 – 698. Nayer, M. (1992). Admission criteria for entrance to physiotherapy schools: How to choose among many applicants. Physiotherapy Canada, 44, 41 – 46. Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions.see comment. Advances in Health Sciences Education, 6(2), Tan, K. P., Meredith, P., & McKenna, K. (2004). Predictors of occupational therapy students clinical performance: An exploratory study. Australian Occupational Therapy Journal, 51(1),