“I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description and perhaps I could never succeed in intelligibly doing so. But I know it when I see it……” 1964 Potter Stewart
Carmichael, Kelly D. MD; Westmoreland, James B. MD; Thomas, John A. MD; Patterson, Rita M. PhD May 2005, Southern Medical Journal The only statistically significant correlations to OITE scores were USMLE step I performance (220) and marital status (married)
Paul J. Dougherty MD; Norman Walter MD; Peter Schilling MD; Soheil Najibi MD, PhD; and Harry Herkowitz MD March 2010, Clinical Orthopaedics and Related Research Those who averaged in the 27th percentile or lower on the OITE had a 57% chance of failing the ABOS Part I examination Correlation coefficient of 0.53 between the USMLE Step 1 and ABOS Part I Average correlation coefficient of 0.50 for postgraduate years (PGY) 2 through 5 OITE scores and ABOS Part I
KLEIN, GREGG R. MD; AUSTIN, MATTHEW S. MD; RANDOLPH, SUSAN; SHARKEY, PETER F. MD; HILIBRAND, ALAN S. MD May 2004, The Journal of Bone and Joint Surgery-American Volume Correlation coefficient for the USMLE Step-I score and the ABOS Part-I percentile score was 0.38 (p = 0.002) Correlation coefficients were also calculated for the OITE percentile scores for years 3 (r = 0.52, p < ), 4 (r = 0.49, p < ), and 5 (r = 0.69, p < ). There was a high risk of failure (63%; five of eight failed) on the ABOS Part-I examination when a resident scored below the twenty-ninth percentile for PGY-3 and below the twentieth percentile for PGY-5
5 programs evaluated 176 current and former residents evaluated Chairs or PD’s asked to rate overall competence and surgical competence along with the 6 core competencies while “modestly blinded” to USMLE, OITE, and ABOS percentiles
A statistical cross comparison of USMLE 1, USMLE 2, OITE 1-5 percentiles, 6 Core Competencies, ABOS percentiles, Surgical Competency and Overall Competency upon graduation was compiled.
Variableby VariableN Spearman ρ Prob> │ρ│ ABOS Part 1 PercentileOITE ABOS Part 1 PercentileOITE <.0001 ABOS Part 1 PercentileOITE ABOS Part 1 PercentileOITE ABOS Part 1 PercentileOITE <.0001
Correlation with ABOS Survey CategoryNSpearman ρ Prob> │ρ│ Patient Care Medical Knowledge650.54<.0001 Practice-Based Learning and Improvement Interpersonal and Communication Skills Professionalism Systems-Based Practice Surgical Competency Overall Competency
Survey/Score CategoryScore CategoryRSp - Value Medical KnowledgeABOS %tile OITE %tile YIT-5ABOS %tile OITE %tile YIT-2ABOS %tile Surgical CompetencyABOS %tile Medical KnowledgeOITE %tile YIT Overall CompetencyABOS %tile Medical KnowledgeOITE %tile YIT Medical KnowledgeOITE %tile YIT OTIE %tile YIT-3ABOS %tile USMLE Step 2ABOS %tile ABOS was more correlative with surgical and overall competency than OITE (medium correlation)
Survey/Score CategoryScore CategoryRSp - Value PBL & ImprovementABOS %tile Systems-Based PracticeABOS %tile ProfessionalismUSMLE Step Interper. & Comm. SkillsUSMLE Step Surgical CompetencyOITE %tile YIT Interper. & Comm. SkillsABOS %tile Medical KnowledgeOITE %tile YIT Surgical CompetencyOITE %tile YIT OITE %tile YIT-4ABOS %tile Overall CompetencyOITE %tile YIT OITE YIT- 4 Appears to be a throw away year
Survey/Score CategoryScore CategoryRSp - Value Surgical CompetencyOITE %tile YIT Surgical CompetencyOITE %tile YIT Surgical CompetencyOITE %tile YIT Surgical CompetencyOITE %tile YIT Surgical CompetencyOITE %tile YIT Surgical CompetencyUSMLE Step Surgical CompetencyUSMLE Step Surgical CompetencyABOS %tile The only objective score to correlate with surgical competency was ABOS I percentile (medium correlation)
Survey/Score CategoryScore CategoryRSp - Value Overall CompetencyOITE %tile YIT Overall CompetencyOITE %tile YIT Overall CompetencyOITE %tile YIT Overall CompetencyOITE %tile YIT Overall CompetencyOITE %tile YIT Overall Competency has no correlation with OITE YIT 1-5
Our “mildly blinded” study of 5 programs, objective measurements have very little correlation with a program director or chair’s opinion of a resident’s overall competency, core competencies outside of medical knowledge or surgical competency upon graduation. The one exception is the ABOS percentile and surgical competency which is mildly correlative.
“When given a choice on who will operate on me or my children, with all other qualities being equal, I would choose the physician with the higher scores….” Steven Frick. June 2012