1. Medical Education Research Toolkit
Karen Hughes Miller, Brian Mavis, Judy Shea, Kirke Lawton
Teri Cameron, Karen Szauter

2. Session Objectives:
Describe resources and tools available to inform and support education research
Define the process for medical research from idea to publication
Identify available funding resources for medical education research

3. Karen Sz
The name slides are placeholders – will delete from the final slidedeck

4. Some opening thoughts:
We are all doing great work –
Do we recognize it as such?
What resources are available to help us move along?
How to we share ideas to build our community?

5. Be curious…

6. Look through a different lens….

7. Taking a Scholarly Approach to All Activities
Clear goals
Adequate preparation
Appropriate methods
Significant results
Effective presentation
Reflective critique
Glassick,CE., Maeroff, GI., Scholarship Assessed Evaluation of the Professoriate. San Francisco, Jossey- Bass 1997

8. The 7 steps of the educational research process.
Read. A lot!
The 7 steps of the educational research process.
Nuthalapaty. Primer on medical education research. Am J Obstet Gynecol 2012.

9. Moving From an Idea to Research Question to Project (…to publication)

10. Professional Development
Gather resources
People
Data
Professional Development
Idea
Funding
Local
Regional
National

11. Professional Development
Gather resources
Collaborators
Mentors
Experts
People
Advance knowledge and skills
Keep data organized
Identify available sources
Reach out for large data sets
Data
Professional Development
Idea
Funding
Local
Regional
National
Direct costs
Time ( salary)

12. Funding
Local – institutional sources of project funding, local foundations
Regional – organizations specific to your region ( examples include foundations, GEA regional grants)
National - larger scale grants/awards from organizations, foundations (GEA national grant, Gold Foundation, DOD, simulation community)

13. WILL NOT INCLUDE THIS SLIDE– just so you can see the transition :
I tried to frame this as things needed to move forward: Kirke – will talk more specifically about large data through the AAMC Brian – moving data to publications Sam – how RIME helps move things forward and the backbone of high quality research Judy – focus on opportunity for professional development

14. Kirke
The name slides are placeholders – will delete from the final slidedeck

15. Kirke B. Lawton, MA, CIPM, CIPT, Sr. Director, Data and Analytics AAMC
AAMC Data Resources
Kirke B. Lawton, MA, CIPM, CIPT,
Sr. Director, Data and Analytics
AAMC

16. Comprehensive Resources Compensation Resources Faculty Information
Operational and Financial Resources
Applicant, Student, Graduate, and Resident Information

18. Comprehensive Resources
AAMC Data Book
Analysis in Brief
Medical School Profile System
Missions Dashboard
Missions Management Tool
Organizational Characteristics Database

19. Compensation Resources
Deans' Compensation Survey
Dean’s Office Staff Compensation Survey
Department/Division Administrator Compensation Survey
Faculty Salary Survey
Group on Faculty Practice Executive Compensation Survey
Mercer-Integrated Health Networks Compensation Survey
Resident/Fellow Stipends and Benefits Survey

20. Women in Medicine and Science Survey and Report
Faculty Information
Faculty Roster
Women in Medicine and Science Survey and Report

21. Operational and Financial Resources
AAMC-SACME Harrison Survey and Report
Compliance Officers’ Forum Survey and Report
Financial Aid Summary Report
Hospital Operations and Finance Surveys and Reports
Group on Business Affairs Metrics Surveys and Reports
Group on Information Resources IT Survey and Report
Group on Resident Affairs Survey and Report
Medical School Finance Tables
Tuition and Student Fees Survey and Reports

22. Applicant, Student, Graduate, and Resident Information
Curriculum Reports
FACTS Tables on Applicants, Students, Graduates, and Residents
Matriculating Student Questionnaire and Report
Medical School Graduation Questionnaire and Report
Medical School Year Two Questionnaire and Report
Post-MCAT Questionnaire and Report
Report on Residents
Student Records System

23. (sorry, some things are just too wonderful to leave out!)
BRIAN
The name slides are placeholders – will delete from the final slidedeck
(sorry, some things are just too wonderful to leave out!)

24. Using AAMC Datasets in Educational Research
Brian Mavis, PhD Office of Medical Education Research and Development

25. Institutional Evaluation
Routine reporting of institutional outcomes
Monitoring impact of:
-- medical school class size expansion
-- community campus expansion
-- curricular change
-- implementation of a new curriculum

26. Institutional Reporting
Historical trends related to …
Matriculating Student Questionnaire
Time since college graduation
Outstanding educational loans
Specialty choice
Intended state of practice
Graduation Questionnaire
Medical school debt
Confidence in clinical skills for residency
Plans to practice in an underserved area
Plans to practice in Michigan
Student satisfaction

27. AAMC GQ: Plan to practice/work primarily in an underserved area

28. Monitoring Change: Class Size
Matriculating Student Questionnaire
Matriculant characteristics
Specific experiences:
research, post-bac program, etc
Medical school choice factors
Career interests
Graduation Questionnaire
Satisfaction with quality of education
Responsiveness of leadership
Mistreatment

29. Monitoring change … Age Sex Race/ethnicity
When decided to study medicine
Lab research
Factors in school choice:
Location
Research
Diversity
Cost

30. Monitoring Change… Career interests: Primary care Underserved Instate
Race/ethnicity
When decided to study medicine
Lab research
Factors in school choice:
Location
Research
Diversity
Cost

31. Using MSQ and GQ
Career interests:
Primary care
Underserved
Instate

32. Scholarship
Published data
Institutional datasets
Customized data

33. AAMC Data Book

34. Section B: Applicants, Students and Graduates

35. AAMC Graduation Questionnaire (GQ)
Academic Psychiatry (2017) 41:

37. Published Data from Graduation Survey
Academic Medicine, Vol. 89, No. 5 / May 2014
pages

38. Medical Student Mistreatment

39. Search of Curriculum Inventory
Three data sources
--survey of deans
--survey of residents and fellows at home institution
--Curriculum inventory search of medical school curricula
Of the 145 U.S. medical schools included in the AAMC Curriculum
Inventory academic year benchmarked data, 82 of the 145
(56.6%) schools were in a state where medical marijuana was legal.
Only 13 (9.0%) of the participating schools documented any medical
marijuana education in their submitted curriculum content. No differences
were seen in inclusion of medical marijuana education in the
curriculum based on legality of medical marijuana in the state; 8 of the
82 (9.8%) medical schools in a state where medical marijuana was legal
documented education about medical marijuana and 5 of the 63 (7.9%)
medical schools in a state where medical marijuana was not legal
documented education about medical marijuana (Chi-square =0.14,
df= 1, p = 0.70).
Drug and Alcohol Dependence 180 (2017) 151–155

40. Medical Marijuana Curricula
Of the 145 U.S. medical schools included in the AAMC Curriculum Inventory academic year benchmarked data:
82/145 (56.6%) schools were in a state where medical marijuana was legal
Only 13 (9.0%) of the participating schools documented any medical marijuana education in their submitted curriculum content
No differences found related to inclusion of medical marijuana education in curriculum based on legality of medical marijuana in the state
8 of the 82 (9.8%) medical schools in states with legal medical marijuana
5 of the 63 (7.9%) medical schools in states where it was not legal

41. Linked Records from MSQ, GQ & Faculty Roster
Adv in Health Sci Educ (2013) 18:279–289

42. Step 1 Failures (First Try) and Career Outcomes
No differences in specialty choice, type of practice or full time faculty appointment. Graduates who initially failed Step 1 were less likely to be board certified at the time of follow-up

43. Linked Records from MSQ, GQ & Faculty Roster System
Academic Medicine, Vol. 85, No. 7 / July 2010
pages

44. Full-Time Faculty Medical School Appointment
Each of the following was independently associated with a greater likelihood of full-time faculty appointment:
Women
Graduation from an MD/PhD program
Higher USMLE Step l score,
AAMC GQ Career Preference of “undecided” or ”full time university faculty”
The choice of either family medicine or surgical specialties was independently associated with a
lower likelihood of full-time faculty appointment.
Each of the following was not independently associated with fulltime faculty appointment:
Race/ethnicity
Year of graduation
Level of debt
Expected extent (at matriculation) to which career will involve research
Type of medical school.

45. Karen (Sam) Miller

46. Research in Medical Education (RIME) Program Selection Committee
Chair :
Karen Hughes Miller, PhD, MEd

47. What RIME Does…
As a part of AAMC Group on Education Affairs (GEA) and a participant in MESRE, the RIME committee selects outstanding papers in medical education research for presentation at the annual Learn Serve Lead conference and for publication in our annual supplement to Academic Medicine.
We also host an invited (plenary) conference presentation, and recently have begun to actively encourage med ed research in socially relevant areas.

48. What RIME Adds to the Toolkit…
The foreword to our 2016 supplement, Characteristics of RIME papers that Make the Cut, is an in-depth analysis of the components of successful research papers.
It is highly descriptive and generalizable to all med ed research.
It’s a good “self review” tool as authors prepare their work.
The foreword to our 2017 supplement, Sticks and Stones: Moving Toward a More Productive Peer Review Process, is a companion to the foreword.
It is designed to help early and mid career researchers understand and value peer-review, AND to help reviewers better fulfill their role.

49. What RIME Adds to the Toolkit…
Our 2017 commentary article, Racism and Bias in Health Professions Education: How Educators, Faculty Developers, and Researchers Can Make a Difference is included in the toolkit because it calls on us to consider a wider range of topics and methodologies in med ed research.
Adding social science research methods to our skill sets can widen our areas of investigation.
Taking on socially relevant topics keeps our research current and useful.
Administrators want make data driven decisions, so let’s step up and provide the data they need.

50. Feedback from RIME Reviews…
Submitting to RIME is not for wimps!
We use a double review process then face-to-face “defense” of articles under consideration. About 12% or submissions are accepted.
However, we have made a concerted effort on the past few years to
Schedule time for brief, focused revisions for articles “on the bubble”.
Consider new, innovative, and non-traditional topics.
Provide specific, constructive feedback to authors.

51. Words of Wisdom… Begin your study with the final publication in mind.
Your IRB proposal should foreshadow your article… this is the ideal time to hone your problem, question, and hypothesis.
Search the literature, then search again! If you cannot put your work into context, it will never see the light of day.
Use citations for both background AND methodology. Reviewers want assurance that your process was valid.
“Limitations” are your friend! Be descriptive and specific. Study limitations set the parameters for your work and structure your next steps.
Read your work out loud. If you turn blue before the end of the sentence, it’s too long.

52. JUDY

53. Medical Education Research Certificate (MERC)

54. What it is:
The Medical Education Research Certificate (MERC) program is intended to provide the knowledge necessary to
understand the purposes and processes of medical education research,
become informed consumers of the medical education research literature
be effective collaborators in medical education research.

55. Workshops Three hour workshops Focus on a key skill or area
Hands-on activities and active participation
Six workshops must be completed to qualify for the certificate.
Data Management and Preparing for Statistical Consultation
Program Evaluation and Evaluation Research
Formulating Research Questions and Designing Studies
Questionnaire Design and Survey Research
Hypothesis-driven Research
Searching and Evaluating the Medical Education Literature
Measuring Educational Outcomes with Reliability and Validity
Scholarly Writing: Publishing Medical Education Research
Introduction to Qualitative Data Collection Methods

56. New/Coming Up
MERC 2.0 – online/distance learning
Robust Evaluation

57. Medical Education Research Certificate (MERC)
MERC Program Activity for 2017 To Date
Type of Venue
# of Sites
# of Workshops
AAMC-related conferences 5 12
Medical school programs 14 63
Other venues 4 23
Total 98
MERC Program Activity: 5 Year Overview
Year
# workshops
# facilitators
# certificates
2013 90 18
122
2014 73 22
101
2015
114 23 48
2016
109
111
2017 YTD 98 20 81