1. Curriculum Inventory Administrators’ Group December 14, 2016
Terri Cameron, MA
Director, Curriculum Programs

2. Agenda
CI / Upload
December CI in Context: Use of Cadavers in US Medical Schools
January CI in Context: Curriculum Structure in US Medical Schools
Presentation: CI Update at Learn Serve Lead 2016
Medical school highlight: University of Pittsburgh
Update on Medical School Schematics page on the CI website
CI for CQI and Accreditation Task Force
CI Research Group
Annual Evaluation of CI Admin Groups
CI Focus Groups at Learn Serve Lead 2016
Next meeting: Wednesday, January 11, 1 pm ET

3. CI in Context: December 2016
Use of Cadavers in US Medical School

4. CI in Context: January 2017
Curriculum Structure in US Medical School

5. Curriculum Inventory Update November 15, 2016
Terri Cameron, MA
Director, Curriculum Programs
Walter Fitz-William, MPP Senior Program Operations Specialist

6. Objectives Share AAMC Update
Provide update on Curriculum Inventory Cycle
Describe CI Advisory Board Review of CI Reports
Focus Groups at Learn Serve Lead
Discuss options for CI Support of CQI and Accreditation
Report activities of CI Research Group
Explain AAMC Data Classification Effect on Curriculum Inventory

7. AAMC Update
Alison Whelan, MD Chief Medical Education Officer

8. Curriculum Inventory Participation
90 Schools
120 Schools (82 of 90 retained; 28 new schools)
135 Schools (114 retained; 21 new schools)
141 Verified
(10 new schools)
8 Schools that participated previously did not participate in due to changes in curriculum management systems, changes in administration, and/or accreditation preparation.
US Medical Schools 85
115
127
134
Canadian Medical Schools 5
US Osteopathic Schools (Pilot)
n/a 3 2

9. Data Validation Outliers in numbers of sequence blocks
Outliers in numbers of events
Duplicate events / sequence blocks
Issue: duplicate events with different names or metadata
Increase in data from previous year (where applicable)
Drastic differences in data from previous year
Future: assign consistent content terminology? Natural language processing?

17. CI in Context: November 2016
Use of Participation as an Assessment Method David W. Musick, PhD. Virginia Tech/Carilion School of Medicine

18. CI in Context: December 2016
Use of Cadavers in US Medical School

19. New Medical School Schematics Site

21. Annual CI Chart Update
LCME AQ II Charts reviewed/updated beginning November 1
Will be adding dynamic links to other resources to all charts (e.g., topic charts)
Will be adding trend data where available
CI Charts updated as part of this process:
School Institutional Objectives/Competences Mapped to Physician Competency Reference Set (PCRS)
Instructional Methods Used in US Medical Schools
Assessment Methods Used in US Medical Schools
Resources Used in US Medical Schools
Existing Content Reports (available to Participating Schools)

22. Review by CI Advisory Board
What do you believe this chart is representing?
Does the title reflect the purpose?
Could you present this chart to your Curriculum Committee and answer questions about what it is saying, where the data came from, etc.?
What enhancements should be made?
What additional explanations are needed?
What can we do to make charts easier to find?

23. Curriculum Inventory Advisory Board
Giulia Bonaminio, PhD Research Professor and Associate Dean for Medical Education, University of Kansas Medical Center Jennifer G. Christner, M.D. Dean, Baylor College of Medicine Colleen Grochowski, Ph.D. Associate Dean for Curricular Affairs, Duke University SOM
Stan Hamstra, PhD Vice President for Milestones Research and Evaluation, ACGME
Nancy Hueppchen, M.D., M.Sc. Associate Dean, Undergraduate Medical Education / Associate Professor, Maternal-Fetal Medicine, Johns Hopkins SOM
Kevin Krane, M.D., FACP Vice Dean of Academic Affairs, Tulane University SOM

24. Curriculum Inventory Advisory Board
John Mahoney, M.D. Associate Dean for Medical Education, University of Pittsburgh SOM Susan Masters, Ph.D. Associate Dean of Curriculum, University of California – San Francisco, SOM Paul R. Standley, Ph.D. Assistant Dean, Curricular Affairs and Professor of Basic Sciences and Physiology, University of Arizona COM - Phoenix
Hugh A. Stoddard, Ph.D., M.Ed. Assistant Dean for Medical Education, Emory University SOM
Ex Officio:
Valerie Smothers, M.A. Deputy Director, MedBiquitous
Marian Taliaferro Director, Reference Center & Archives, AAMC

25. Focus Groups at Learn Serve Lead
Friday, November 11, Convention Center 302:
Standardized Vocabulary: 10 am PT
CI Data for Data Warehouses: 11 am PT
CI Data Classification: 1 pm PT
CI and Educational Research: 3 pm PT
CI Support of CQI and Accreditation Documentation: 4 pm PT
Clerkship Documentation: 5 pm PT

26. CI Support of CQI and Accreditation
End result: User Guide for Documenting for CQI and Accreditation in Your Curriculum Management System
Development of synonyms for content data in DCI
Discussion of issues from schools:
What were your greatest challenges in responding to questions/elements related to curriculum content in the DCI?
What were your solutions to those challenges?
As you review your CI Verification Report, do the ‘DCI’ tables look like they would if you filled them in by hand? If not, why not?
What can we add to the Verification Report to help with CQI and accreditation?

27. Curriculum Inventory Support of CQI and Accreditation Task Force
Meghan Burke Asik Weill Cornell Medicine Jennifer Christner, MD Baylor University College of Medicine Naomi Lacy, PhD Texas Tech University Health Sciences Center Paul L. Foster School of Medicine Alan Lesse, MD Jacobs School of Medicine and Biomedical Sciences at the University at Buffalo Jose Lopez Texas Tech University Health Sciences Center Paul L. Foster School of Medicine Cindy Lybrand, MEd East Tennessee State University James H. Quillen College of Medicine
David Rachleff, MSW University of California – San Francisco School of Medicine
Michael Schwartz, PhD
Yale University School of Medicine
Vicki Park, PhD University of Tennessee Health Science Health Science Center
Santiago Toro-Posada, MD
University of British Columbia Faculty of Medicine
Terri Cameron, MA
AAMC
Walter Fitz-William, MPP

28. Curriculum Inventory Research Group
Have begun detailed review of projects and data needs
Strategizing for collaboration with other ‘data groups’
Will continue to meet monthly through March; begin quarterly meetings in April to allow project teams to work on projects and report back quarterly

29. Curriculum Inventory Research Group Members
Jorie Colbert-Getz, PhD University of Utah SDRME Steven J. Durning, M.D., PhD, FACP Uniformed Services University NEGEA Carolyn DuFault, PhD Washington University CGEA Rachel Ellaway, PhD University of Calgary Canada Jeanne Farnan, MD, MHPE University of Chicago CGEA Colleen Grochowski, PhD Duke University SGEA Tanya Horsley, PhD Royal College of Physicians RIME
Brian Mavis, PhD Michigan State University CGEA
Martin Pusic, MD, PhD New York University MESRE
Arianne Teherani, PhD Univ of California – San Francisco WGEA
Terri Cameron, MA
AAMC
Walter Fitz-William, MPP
Katherine McOwen, MA AAMC

30. CI Data Classification Questions for Consideration
Before today, what was your understanding of the confidentiality of Curriculum Inventory Data?
Do you see value in allowing faculty to find potential collaborators for research, publication and presentation? If not, why not?
Do you see value in allowing researchers to include institution names in research datasets, as long as the research is reviewed and deemed appropriate by AAMC, and a Data Grant Release makes clear how the data can be used?
As medical education researchers, would you need to know the names of the schools included in curriculum data?

31. CI Data Classification
The following are statements from Educational Researchers. Do you agree or disagree with each statement? Why or why not?
If school names were included, the CI would have more accountability for better, cleaner data.
Schools that participate in the CI are part of an academic commons that should support sharing data in a manner that can be analyzed.
Including institution names as part of a research dataset is part of being in an academic environment that supports improvement in medical education.
Research datasets are more valid with more participating schools.
With today’s environment (social media) students are sharing more than would be available from curriculum data.
School administrators and curriculum committees cannot do appropriate information management without knowing what schools are doing.
Quality improvement is often based on single context and supposition; the CI provides a large dataset that allows for more thoughtful approach and less guessing.
Including institution names as part of a research database could enable educational outcomes research.

32. CI Data Classification
Has your school ever requested CI data with institutional names?
What was the purpose?
If not, can you think of an example of why you might want to make such a request?
Can you think of examples where it might be important for large- scale educational research where including institution names would be essential?
Will you still participate in the Curriculum Inventory if there is a process that would allow release of data with institution names in appropriate circumstances? If not, why not?

33. CI Data Classification: Message on CI Portal Home Screen
Welcome to the AAMC’s Curriculum Inventory season. Your voluntary participation is greatly appreciated as it contributes to the growth and development of a centralized database of curricula, including content, structure, delivery, and assessment, as well as furthering medical education research and helping us present a chorus of voices from the diversity of our members. The AAMC classifies the data collected in the Curriculum Inventory as unrestricted and, when appropriate to support medical education and benchmarking efforts, the data may be released with institutional identification. The information provided by your medical school will be handled and protected by the AAMC in secure databases with limited access by key AAMC staff. Please provide data for the Curriculum Inventory by September 30th. If you have questions about participating in the Curriculum Inventory, please send an to

34. CI Data Classification: Example Data Grant Release
AAMC grants this permission with the following conditions:
[Institution] may only use the Data in connection with the “Project.”
This grant is a one-time, non-transferable grant; and
The Data may not be altered in any way and must be accompanied with the following statement: AAMC: [AAMC storage location], [FILE name], as of [Date Extracted]. [FILE] last updated [DATE FILE UPDATED].
In the event a publication results from use of this Data, AAMC requests that a copy of the published work be ed to for our records.
In the event the [Institution] breaches any one of these conditions this grant will be deemed to be immediately revoked by the AAMC and [Institution] may no longer include the Data in their report.
AAMC PROVIDES THESE DATA “AS IS” WITHOUT WARRANTY OF ANY KIND AS TO COMPLETENESS OR ACCURACY, AND [INSTITUTION] RELEASES AAMC FROM ANY AND ALL LIABILITY FOR INACCURATE OR INCOMPLETE INFORMATION.
By accepting the Data, [institution] accepts the terms of this grant.

35. Proposals for Spring/Summer Education Meetings
Focused on two areas:
Best Practices for Using Curriculum Management Systems to Support Curriculum Committees, CQI, Faculty, Administrators, Staff, and Students
Use of the CI for Educational Research / Resources for Educational Research
Group on Information Resources (GIR) is a joint meeting with the Group on Business Affairs (GBA) and Group on Institutional Planning (GIP)
Offers great opportunity for expanding knowledge and utility of CI and highlighting advantages of local curriculum management systems

36. Plans for the Coming Year:
Charts moved to ‘Participating Schools’ area (will require login)
Improved search function
Expansion of resources to all charts
Data Quality Reviews / Follow-up
Enhanced opportunities for CQI
User Guide for Documenting Curriculum Data for CQI and Accreditation

37. Thank you for your support!
Questions?
Thank you for your support!

38. CURRICULUM METADATA: VITAL SIGNS FOR CURRICULUM MANGEMENT
John Mahoney, MD
Associate Dean for Medical Education
University of Pittsburgh School of Medicine

39. Curriculum Management LCME Compliance Tracking
OVERVIEW
Curriculum Mapping
Curriculum Management
LCME Compliance Tracking

40. Instructional & Assessment Methods, Learning Objectives
METADATA TYPES
Intrinsic & Obvious –
Instructional & Assessment Methods, Learning Objectives
Novel & Hot Topics –
Local Priorities,
LCME 7.1
Manual Tagging vs. Automated
Tagging Accuracy & Consistency – will determine usefulness of all metadata

41. Map is just a way to connect the dots
Curriculum Mapping
Map is just a way to connect the dots
Example: link L.O. to content duration or placement
Ability to view close-up or zoom out
Practical uses different depending on role/point of view

42. Curriculum Management
Overlaps, builds on mapping concept
Knowing what, where, how much is necessary for making decisions
If the mapping or metadata reports are too complex, limits use to “experts”
Not necessarily used every day

43. SEARCHING & COMPLIANCE TRACKING
Carefully selected tags can ease compliance tracking
Tagged metadata potentially more valuable than plain search results
Topics & Themes that target school priorities
Hottest topics – new or undergoing change

44. Sample Special Tags Pathology Opioids Pharmacology Social Medicine
Oncology
Global Health
Nutrition
Interprofessional
Pain
EPAs

45. Contact
John Mahoney

46. CI 2015-2016 Data Issues / Planning for 2016-2017
Typos
Use of abbreviations without explanations
All of the medical school's sequence blocks are linked to the same events.
The average, minimum, 25th percentile, and median number of events per sequence block is the same, but the maximum number is different.
The average, minimum, 25th percentile, and median number of events per sequence block is the same, and the maximum number is equal.
The medical school has submitted CI data more than once, and has not provided more academic years than Academic Year 1 in the recent year.
The medical school has submitted CI data more than once, and has provided fewer academic levels than in the previous submission.
The medical school's sequence blocks are more than 8 years apart.
Content exceeds 3900 characters in a 4000-character limit field – data is often truncated.

47. CI Administrators’ Group Annual Evaluation
Being developed now for distribution in January
Deadline two weeks after distribution
Results at February session
Make changes as appropriate beginning in March

48. CI Administrators’ Group Annual Evaluation
I gained valuable knowledge and skills during these webinars.
I will apply new knowledge and skills that I learned during these webinars in my professional role.
I would recommend these webinars to a close colleague.
Overall, I was completely satisfied with this webinar.
I find the following updates to be most useful:
AAMC Updates
CI Upload Process, Tips
CI Website Updates
CI Requested Report Updates
CI in Context
Report of the Month
Presentations from educational conferences
Medical School Highlight
What updates should be added? _____________________________
The frequency of these webinars should be __ monthly __ bi-monthly __ quarterly

49. CI for CQI and Accreditation (CICA)
Update:
Determined not to include schematics for 7.0 as part of the development process.
Determined that 7.2.a still merited work, even though direct references to the content of 7.0 would no longer be possible and participants would provide examples of their own responses for examination against the content of recent submissions to the Curriculum Inventory.

50. Curriculum Inventory Research Group
Focusing on three projects for 2017:
Review of Longitudinal Integrated Clerkships / Clerkship Continuity Experiences
Longitudinal Study of how Boot Camps / Transition to Residency / Individual school measurements prepare students for residency
Review of CI aggregate submissions for schools that implemented new curricula in 2016 – can we find new trends and initiatives from the data?

51. Curriculum Inventory Focus Group: Standardized Vocabulary
Are you aware of the CI Standardized Vocabulary?
Why do you think we have a Standardized Vocabulary for Instructional Methods, Assessment Methods, and Resources?
Do you see the value in documenting resources?
Is your school documenting resources? Why or why not?
Can you find the Standardized Vocabulary easily?
How can we make it easier for you to find?
Does the CI Standardized Vocabulary meet your needs as you document your curriculum? Why or why not?
Do you know how to submit a ‘challenge’ to the Vocabulary?
How often do you think the Vocabulary should be updated?
Continued to next slide

52. Curriculum Inventory Focus Group: Standardized Vocabulary
What else would you like to tell us about the Vocabulary for Instructional Methods, Assessment Methods, and Resources?
Are you using a content vocabulary to document or map your curriculum?
Who developed your content vocabulary?
How did you develop your content vocabulary?
Does it meet your needs? Why or why not?
How often is it reviewed?
Would your school support incorporation of ‘national vocabulary’ as part of your school’s content vocabulary?

53. Curriculum Inventory Focus Group: CI Data for Data Warehouses
Share basic concept of ‘cloud data’
 Gather information about:
If your school had access to this data, how would the data be used?
What resources does your school have or need to work with data?
What support would be necessary for your school to use this data?
Does your school have a data warehouse?
Does your school have data visualization software?
How would your school prefer the data to be organized / accessed?
What vendors / software / systems is your school currently or considering working with?
Do you see this as a positive move in making data available to schools? Why or why not?

54. Curriculum Inventory Focus Group: CI and Educational Research
Do you know how to access Curriculum Inventory Data for research projects?
What types of medical education research projects do you think the CI can support?
Do you have ideas for medical education research in mind?
Would it be helpful to you if the CI could help you identify potential partners for collaboration on medical education research projects?
Can you provide examples of current use of CI data to meet individual schools' needs for research, benchmarking, or CQI?
(continued to next slide)

55. Curriculum Inventory Focus Group: CI and Educational Research
Can you provide examples of institutional-specific application of CI data for individual faculty needs?
What data do you think the CI currently collects?
Is there additional data that the CI should collect?
Thinking broadly with the potential of including other datasets (MCAT, Matriculation Questionnaire, Graduation Questionnaire, NBME data, ACGME data, IPEC data, etc., what types of research projects do schools need to support their curriculum efforts?
Do you have suggestions for how we can begin to work with other groups to build a research database to support these ‘Big Data’ projects?

56. Curriculum Inventory Focus Group: CI Support of CQI and Accreditation Documentation
Development of synonyms for content data in DCI
Issues / Resolutions
End result: User Guide for Documenting for CQI and Accreditation
Issues from schools:
What were your greatest challenges in responding to questions/elements related to curriculum content in the DCI?
What were your solutions to those challenges?
As you review your CI Verification Report, do the ‘DCI’ tables look like they would if you filled them in by hand? If not, why not?
What can we add to the Verification Report to help with CQI and accreditation? Continued to Next Slide

57. Curriculum Inventory Focus Group: CI Support of CQI and Accreditation Documentation
Share basic concept of ‘cloud data’
Gather information about:
How schools would prefer data to be organized / accessed
What resources schools have / need to work with data
What vendors / software / systems schools are currently or considering working with
How schools would use the data and what support they would need for those uses

58. Participate in WebEx Focus Groups
10 People per group
Mid-to-late January
1 hour WebEx
Send note to to participate

59. Next meeting: Wednesday, January 11, 1 pm ET
(Second Wednesday of each month, 1 pm ET)
Registration Links posted in Training and Resources section of
Please send agenda items to