1. Session #2 Fairness and the New MCAT Exam

2. AAMC Admissions Hub www.aamc.org/admissions
Resources on Fairness
FACTS data on MCAT Scores (aamc.org/facts)
Using MCAT Data in 2016 Medical Student Selection
“Do Racial and Ethnic Group Differences in Performance on the MCAT Exam Reflect Test Bias” (Academic Medicine, May 2013)
Khan Academy collection and other free and low cost test preparation resources

3. Learning objectives By the end of this session, you will:
Know how fairness was key to the design and development of the new exam
Learn about local efforts to help students prepare for the new exam

4. Refer to information in this guide during the session

5. Fairness played a critical role in blueprint development

6. Fairness played a critical role in blueprint development
Balance academic competencies from the natural, behavioral, and social sciences
Test psychology and sociology concepts like discrimination, stereotype threat, and socio-economic inequalities
Increase attention to population health, studies of diverse cultures, ethics, philosophy
Test concepts covered widely in undergraduate science courses

7. Fairness played a critical role in blueprint development
Longer sections that allow medical schools to consider applicants’ strengths and weaknesses in relation to their curricula
More working time per question, allowing test takers to work through the information presented in the passage, think about the question, and decide on the best answer

8. Fairness played a critical role in developing test questions

9. We used tried-and-true procedures to develop the new questions, and some new ones as well
Editorial review
Technical review
“Summits” with experts
Bias and sensitivity review
“Field testing” – try out questions to see how they work

10. Addressing Fairness when Developing Questions for the Psychological, Social, and Biological Foundations of Behavior Section

11. Potentially sensitive content
Some content in the Psychological, Social, and Biological Foundations of Behavior Section is potentially sensitive
Prejudice and Bias
Discrimination
Health Disparities
Healthcare Disparities

12. Gathering feedback on potentially sensitive content
Does potentially sensitive Psychological, Social, and Biological Foundations of Behavior section content interfere with the performance of examinees from underrepresented minority groups by distracting them or slowing them down?

13. Methods of gathering feedback
Focus Group with Medical Students and MCAT Exam Editors
Review of Responses to PSBB Study Post-administration Survey
Interviews with 2013 MCAT Examinees

14. Results Very few individuals identified the content as distracting
Individuals underrepresented in medicine were no more likely to identify the content as distracting than others
In general, participants noted that topics such as health disparities are relevant to medical school and important for future medical students to know

15. Monitoring Fairness Data During Field Testing

16. New test questions were tried out on examinees in 2013 and 2014
Examinees volunteered to take a 32-item “trial section” at the end of the test
Volunteers who showed a “good faith effort” received $30 and feedback on the new questions
Data from the trial section were used to:
Identify questions with good statistical properties
Highlight any opportunities for improved item development

17. We monitored the field test data for fairness
Attended a minority-serving institution or an institution with no pre-health advisor
Received fee assistance from the AAMC
Males vs. females
Race/ethnicity (white, Black, Latino)

18. Results More than 113k students tried out new questions
Participation rates were high, and the vast majority of examinees showed a good faith effort
The participants were similar demographically to the population of examinees
The high data quality helped build a large item bank for the new MCAT exam
Differences in average performance were comparable to the differences on the old MCAT exam

19. After Lunch, You’ll See Early Data from the April and May 2015 Examinations

20. Fairness played a critical role in developing the score scales for the new MCAT exam

21. Developing the Score Scales with Fairness in Mind

22. Many data sources influenced the design of the new score scales
Surveys of admissions officers on the use of MCAT scores and other applicant data in admissions
Acceptance rates for applicants with different ranges of MCAT scores and undergraduate GPAs
Graduation rates for medical students with different ranges of MCAT scores and undergraduate GPAs

23. Medical schools use academic metrics, experiences, and attributes holistically
, and demographic data. 3

24. Acceptance Data

25. 44% of applicants received one or more acceptances
Percentages of applicants admitted into one or more medical schools
44% of applicants received one or more acceptances 4

26. Percentages of 2012-2014 applicants admitted into one or more medical schools
9% of applicants with GPAs above 3.8 and MCAT scores above 38 are not admitted into any medical schools 4

27. Percentages of 2012-2014 applicants admitted into one or more medical schools
8% of applicants with GPAs and
MCAT scores are admitted into at least one medical school 4

28. Medical Student Performance

29. The vast majority of medical students graduate in four years
87% of medical students graduate in 4 years 44

30. Medical students with a wide range MCAT scores and UGPAs graduated in 4 years
81% of medical students entering with MCAT scores between graduate in 4 years 44

31. A new exam means new score scales
Decision Rules
No overlap
Emphasis on the center of the scale

32. 14

33. After Lunch, You’ll Have a Chance to Work with the New Score Reports

34. Our Research on Fairness is Just Beginning

35. MCAT validity research
The MCAT Validity Committee is responsible for evaluating the new exam and how well it:
Supports diversity and is fair
Predicts medical students’ academic success
Helps admissions officers do their work

37. Validity committee members
Catherine Lucey, M.D. (Chair)
Aaron Saguil, M.D., LTC (Vice-chair)
Ngozi Anachebe, M.D.
Ruth Bingham, Ph.D.
Barbara Beckman, Ph.D.
Kevin Busche, M.D.
Victoria Cannon
Steven Case, Ph.D.
Deborah Castellano, M.S.
Jerry Clark, Ph.D., M.B.A.
Julie A. Chanatry, Ph.D.
Daniel M. Clinchot, M.D.
Liesel Copeland, Ph.D.
Hallen Chung, M.A.
Martha L. Elks, M.D.
William Gilliland, M.D.
Jorge A. Girotti, Ph.D., M.A.
Joshua Hanson, M.D., M.P.H.
Brandon Hunter
Loretta Jackson-Williams, M.D.
David Jones, Ph.D.
R. Stephen Manuel, PhD.
Stephanie C. McClure, M.D., FACP
Janet McHugh 24 37

38. Validity committee members
Stephen Nicholas, M.D.
Wanda Parsons, M.D., FCFP
Boyd F. Richards, Ph.D.
Stuart Slavin, M.D.
Jenna Smith, Ph.D.
Aubrie Swan Sein, Ph.D., Ed.M.
Doug Taylor
Carol Terregino, M.D.
Ian W. Walker, M.D.
L. James (LJ) Willmore, M.D.
Robert A. Witzburg, M.D.
David Wofsy, M.D. 24 38

39. The MCAT validity research will evaluate a number of questions, for example:
Diversity and fairness
Are differences in average scores for test takers in different racial/ethnic groups smaller or larger than differences on the old exam?
Academic preparation
Will more humanities and psychology majors take the new exam?
Admissions decision making
How are medical school admissions committees using scores from the new exam in admissions?
Predicting medical student performance
Will scores from the new exam predict academic performance throughout medical school?

40. Two working groups are studying fairness issues
Questions
Access to preparation resources
Access to courses
Barriers to success
Changes to diversity
Other
People
Pre-health advisors
Admissions
Diversity affairs
Other
Data
MCAT registration
PMQ
AMCAS applications
Other
Change to test
Changes to preparation
Potential differences in access
Potential increases in group differences

41. We will report validity data as they become available
Applications for the 2016 class sent to medical schools
Predictive validity results for M1 students reported at AAMC Annual Meeting
Predictive validity results for M1, M2 students reported at AAMC Annual Meeting
MCAT2015 launches
MCAT session at the PDC, preliminary predictive validity results for PSBB
2016 class matriculates
Acceptance data reported for 2016 class
Acceptance data reported for 2017 class
June 2015
Nov. 2017
Nov. 2018
April 2015
July 2015
Aug 2016
Jan. 2017
Jan. 2018 25

42. Fairness played a critical role in developing preparation resources for aspiring physicians

43. Helping Students Prepare
Understand
Practice
Study

44. Helping Students Prepare
Understand
Practice
Study

45. Helping Students Prepare
Understand
Practice
Study

46. Helping Students Prepare
Understand
Study
Yesterday I told you about the interactive test content outline. That was new frontier for us. But we weren’t done being bold.
We knew folks were worried about the PSBB section, and the burden that 2 extra courses, one in psychology and one in sociology, would place, disproportionately on the disadvantaged. And MR5’s research caused them to conclude it really wasn’t necessary to add two courses; students could learn this introductory content in a number of different ways.
So AAMC reached out to test publishers of psychology and sociology textbooks. We found partners willing to cross reference their textbooks against the PSBB blueprint. Any student can go online, download this list, and find the chapter in a textbook where they can learn about a specific topic. In this case, knowledge is power. And for those of you worried about the cost of another textbook, some of these resources are free, open access.
Finally, later this year one of the publishers is coming out with a textbook that integrates the foundational concepts from psychology, sociology, and biology that are important building blocks for the medical school curriculum.

47. Khan Academy tutorials teach concepts targeted by the new test

48. What are your colleagues doing?
Jorge Girotti
Ngozi Anachebe
Jorge and Ngozi are going to share efforts at their institutions to help pre-health students get ready for the new exam (each has 5 minutes to share their stories) 48

49. Activity 2.2 Activity 2.2: Table discussion
Participants will have a discussion about solutions they’ve or their colleagues have developed at their institutions to help pre-health students prepare for the new exam.
Take some time at your table to discuss the ways that you and your colleagues are helping students get ready for the new exam.