1. Curriculum Inventory for Curriculum Deans and Administrators
Terri Cameron, MA Director, Curriculum Programs
Walter Fitz-William, MPP

2. We will cover: Why Upload to the Curriculum Inventory?
MedBiquitous Standard
Standardized Vocabulary
Physician Competency Reference Set (PCRS)
XML Export
Data Upload Processes
Data Verification Report
Documenting Multiple Iterations
Documenting Clerkships
CI Upload Process
Data Validation Issues
XML explained in a few moments

3. What Is the Curriculum Inventory?
Streamlined international curriculum data repository
Not a local curriculum management system
No data entry screens – all data is uploaded from local curriculum management systems
Provides graphical interpretations of aggregate and historical curriculum-related data (includes LCME A/Q Part II data)
Serves as the premier source for benchmarking and educational research in medical education
Curriculum Inventory and Reports is an international curriculum data repository and an enhanced reporting tool that streamlines and simplifies data collection for benchmarking and educational research in medical education in the US and Canada. While the Curriculum Inventory is being built, a subset of the Curriculum Reports are currently available and are based on the LCME Annual Questionnaire Part II data. The curriculum reports are graphical interpretations of aggregate and historical curriculum-related data and represent 100% of U.S. medical schools.
Curriculum Inventory & Reports Projected Launch Date: January 2014

4. Why participate?
The Curriculum Inventory is the culmination of a long history of the AAMC collecting information about medical education content, structure, and pedagogy to support:
Medical Education Research (medical school faculty and external researchers)
Continuous Quality Improvement / Benchmarking
Support of Curriculum Committees
Reporting on new trends
Responding to legislative inquiries or providing data when important issues are being discussed
Respond to media inquiries
The Curriculum Inventory is only effective when the data represents most, if not all, medical schools.

5. Why participate?
CI data can be used to assist in accreditation preparation:
Verification Report includes how CI data looks when used to populate LCME Summary Tables related to curriculum
CI for CQI and Accreditation (CICA) Task Force is creating a set of Best Practices for documenting curricula in curriculum management systems and assisting us in setting up reports that schools can use to respond to other elements and questions related to curricula. This information can be used for continuous quality improvement AND accreditation
Only participating schools can request detailed Curriculum Inventory Reports

6. Curriculum Inventory Participation
90 Schools
120 Schools (82 of 90 retained; 28 new schools)
135 Schools (114 retained; 21 new schools)
141 Verified
(10 new schools)
8 Schools that participated previously did not participate in due to changes in curriculum management systems, changes in administration, and/or accreditation preparation.
US Medical Schools 85
115
128
134
Canadian Medical Schools 5
US Osteopathic Schools (Pilot)
n/a 3 2

7. How can the CI help my school?
Educational research
Trends in medical education
Benchmarking
Tracking competency based education
Graphical interpretations of aggregate and historical curriculum-related data
All schools (US and/or Canadian)
By healthcare profession
By country / geographical region
By level of training (UME / GME)

8. https://www.aamc.org/initiatives/cir/curriculumreports 60+ Reports –
Use search feature to find charts.
Detailed CI Reports are not included in this list – they are only available to Participating Schools. A list has been strategically hidden in the middle of this page (we are working on updates for the site) – send requests for the reports to

9. Curriculum Inventory Reports
Two sources of data:
LCME Annual Questionnaire Part II
100% of US Medical Schools
Institutional and policy data, as well as ‘overview’ curriculum data
Curriculum Inventory
92% of US Medical Schools
29% of Canadian Medical Schools
Detailed curriculum data
The two sources complement each other

10. Curriculum Inventory Reports
Curriculum Inventory data will augment LCME Annual Questionnaire data with details such as:
Number of courses where content is covered (total, mean, median, range)
Number of events where content is covered (total, mean, median, range)
Instructional and assessment methods used in events where content is covered
Resources used in events where content is covered
Competencies linked to events and courses where content is covered
The ultimate goal for all Curriculum Inventory Reports is to include options to: 1) download the graphic in various formats; 2) download the aggregate dataset; 3) provide easy access to related reports; and 4) provide easy access to other AAMC resources related to the topic, including MedEdPORTAL publications. When fully implemented, charts like this one, which is based on LCME Annual Questionnaire Part II data, would be augmented with details from Curriculum Inventory data to provide details such as the number of courses and events where this content is taught, the detailed instructional and assessment methods used in the events, resources used in the events, and competencies linked to the events and courses.

11. Curriculum Inventory Reports are accessed over 1,250 times per month
Accessed over 1,000 times in 2015
Jan 2015
Feb 2015
Mar 2015
Apr 2015
May 2015
Jun 2015
202
102
149 92 84 70
Jul 2015
Aug 2015
Sep 2015
Oct 2015
Nov 2015
YTD Total 83 79 99 57 38
1055
The Curriculum Inventory reports are regularly sought after – while these data are somewhat dated, this is still one of our top requested reports.

12. Drop-down includes ONLY participating schools
needs to be from your institution
Options to include data from:
All Schools
US MD
CA MD
When you request a specific CI report, you will find yourself faced with this form. Only participating schools can request forms, and we will need you to use your institutional address. You can specify the keywords and concepts you want us to search the Curriculum Inventory for, and then indicate which school types you want us to include in the report. We’ll be in contact with you to work out the specifics of your terms and then we’re off to the informational races.

13. This is an example of parts of a content search report
This is an example of parts of a content search report. You can even view the whole of this in the March 2017 edition of Curriculum Inventory in Context

14. CI in Context: May 2017
Tackling the Opiate Epidemic Through Educational Innovation and Partnership
Melissa Fischer, MD Med, University of Massachusetts Medical School

15. Terminology

16. Terminology
Previous Academic Year (LCME model – if current year is , previous academic year is )
Academic Level (formerly called ‘year of curriculum’)
Sequence Block (Course, Module, Unit, Block, Clerkship)
Nested Sequence Block (Optional) (a way to show a variety of organizational methods, including an ordered set of courses within a phase of instruction, a series of electives, selectives, and parallel blocks)
Integration Block (Optional) (a way to describe how content related to a specific topic, discipline, or theme is incorporated or integrated across Phases, Sequence Blocks, and other curricular structures)
Tracks (Optional)
(a way to document tracks that meet the LCME definition of tracks)
Event (Instructional or Assessment Session) (e.g., Lecture, Lab, Discussion, Assessment)
Nested Sequence Blocks: Sequence blocks are optional. They are a way to show a variety of organizational methods, including an ordered set of courses within a phase of instruction, a series of electives, selectives, and parallel blocks. For example, a Sequence Block called Longitudinal Clerkships might include nested Sequence Blocks called Family Medicine, Ambulatory Medicine, and Chronic Care. Another example might be a Sequence Block called Foundations in Medicine that includes nested Sequence Blocks called Human Behavior, Community/Preventive Medicine, Community Service Learning, Medical Ethics, Medical Interview, Nutrition. The use of Nested Sequence Blocks provides flexibility to set up individual course structures and assessments under the umbrella of a larger structure and assessment.
Integration Blocks: Integration Blocks are optional. They are a way to describe how content related to a specific topic, discipline, or theme is incorporated or integrated across Phases, Sequence Blocks, and other curricular structures. For example, content related to themes, topics, and / or disciplines that a school considers to be the major lynchpins of the curriculum (e.g., anatomy, medical informatics, genetics, geriatrics, etc.,) can be easily tracked across the curriculum using Integration Blocks.
Tracks: A parallel program of study for a subset of the medical student body that requires participating students to complete specific programmatic learning objectives (e.g., in research, primary care, leadership) in addition to the medical educational program objectives required of all medical students.

17. Terminology, Continued
Standardized Vocabulary (Instructional Methods, Assessment Methods)
PCRS (Physician Competency Reference Set)
Expectations
(Competency / Objective / Milestone / EPA)
Verification Report (Curriculum Inventory data report used to verify curriculum details)
MedBiquitous (
XML (Extensible Mark-up Language: similar to HTML (Hypertext Markup Language) used to generate web sites, XML is used to exchange data between systems)
Terry will review.

18. Mission: To develop and promote technology standards for the health professions that advance lifelong learning, continuous improvement, and better patient outcomes. 
Not-for-profit, member-driven, standards development organization

19. CI Standardized Vocabulary
Use local terms for institutional data entry and reporting; match to standardized vocabulary to upload to CIR for aggregate reporting
Instructional Methods
Assessment Methods
Resources
Detailed descriptions available at
Two versions – one showing changes from 2012 Version; one without mark-up.

21. Matching Local Terminology to CI Standardized Terminology
Local Terminology: Instructional Methods
CI Standardized Terminology
Clinical Skills
Clinical Experience - Outpatient
Procedure Training
Clinical Experience - Inpatient
Resource Session
Lecture
Standardized Patient
Simulation (with Resource: Standardized Patient)
Wrap-Up Session
Discussion, Small Group
The MedBiquitous Curriculum Inventory Standardized Terminology Subcommittee reviews the terminology each year, based on literature and terminology schools report as difficult to match (challenges)
Submit matching ‘challenges’ to or post comments in iCollaborative
Matching is school-specific: at some schools, Clinical Skills might be matched to the above example shown in the slide; at other schools, Clinical Skills might be done as a demonstration or simulation. Asking faculty to match to the CI Standardized Terminology at the event level would likely provide more clarity in matching, but that may not be possible at all schools and the matching may need to be done at the program level by a Curriculum Committee, Curriculum Dean, or Curriuclum Staff.

22. Physician Competency Reference Set (PCRS)
Competencies vs Objectives
Competencies, Learning Objectives, Milestones, and Entrustable Professional Activities (EPAs) are all separate concepts with explicit definitions and purposes.
For the Curriculum Inventory, they are all “Expectations”.
The category element can be used to distinguish whether an Expectation is a learning objective, competency, etc., but there are no requirements to use that option at this time.
Competencies
Milestones
Objectives
EPAs
Expectations

23. Physician Competency Reference Set (PCRS)
Partial URI
Physician Competency Reference Set (AAMC 2013)
aamc-pcrs-comp-c0100
PATIENT CARE: Provide patient-centered care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health
aamc-pcrs-comp-c0101
Perform all medical, diagnostic, and surgical procedures considered essential for the area of practice
aamc-pcrs-comp-c0102
Gather essential and accurate information about patients and their condition through history-taking, physical examination, and the use of laboratory data, imaging, and other tests
aamc-pcrs-comp-c0103
Organize and prioritize responsibilities to provide care that is safe, effective, and efficient
aamc-pcrs-comp-c0104
Interpret laboratory data, imaging studies, and other tests required for the area of practice
aamc-pcrs-comp-c0105
Make informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to-date scientific evidence, and clinical judgment

24. Matching School Expectations to PCRS
Program-Level (School) Expectations must be mapped to PCRS competencies
May NOT be mapped to PCRS domains
‘Other’ provided for instances where there is not a direct match between expectations
Event- and Sequence Block-Level Expectations may NOT be mapped to PCRS
Event and Sequence Block Expectations may be mapped to each other or to Program Expectations or not at all
This mapping is pulled from your curriculum management system
PCRS
Program
Sequence Block
Event
For PCRS to work as an aggregate tool, schools should map Program Level Expectations to PCRS competencies. Mapping to domains does not provide a specific enough match for reporting purposes. For those competencies that truly cannot be mapped to a PCRS competency, an ‘Other’ Competency is provided under each PCRS domain.
Because Event and Sequence Block Level Competencies are school specific and often number in the thousands, they should NOT be mapped to PCRS. If they are mapped, they should be mapped to Program-Level Competencies. The CI will be able to track PCRS down to the Event level for reporting purposes based on mapping from Event and /or Sequence Block to Program Level Competencies.

25. Matching Program Objectives to PCRS (Crosswalk)
2. Knowledge for Practice: Demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social-behavioral sciences, as well as the application of this knowledge to patient care.
2.1 Demonstrate an investigatory and analytic approach to clinical situations
2.2 Apply established and emerging bio-physical scientific principles fundamental to health care for patients and populations
2.3 Apply established and emerging principles of clinical sciences to diagnostic and therapeutic decision-making, clinical problem-solving, and other aspects of evidence-based health care
2.4 Apply principles of epidemiological sciences to the identification of health problems, risk factors, treatment strategies, resources, and disease prevention/health promotion efforts for patients and populations
2.5 Apply principles of social-behavioral sciences to provision of patient care, including assessment of the impact of psychosocial and cultural influences on health, disease, care-seeking, care compliance, and barriers to and attitudes toward care
2.6 Contribute to the creation, dissemination, application, and translation of new health care knowledge and practices
2.99 Other knowledge for practice
Demonstrate a broad working knowledge of the fundamental science, principles, and processes basic to the practice of medicine.
2.1 Understand the clinical relevance of scientific inquiry.
2.2 Apply this knowledge in a judicious and consistent manner to prevent common health problems and achieve effective and safe patient care.
2.3 Demonstrate the ability to evaluate emerging knowledge and research as it applies to diagnosis, treatment and the prevention of disease.
2.4 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of individuals.
2.5 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of populations.
2.6 Understand the indications, contraindications, and potential complications of common clinical procedures.
2.7 Perform the basic clinical procedures expected of a new PGY-1.

26. Sequence Blocks, Events, and Expectations Mapping
Some schools map relationships between Event and Sequence Block Expectations
Some schools map relationships between Program and Sequence Block Expectations OR Event Expectations
Some schools do not have any mapping for Sequence Block Expectations OR Event Expectations
The MedBiquitous Competency Framework supports any of these options
PCRS
Program
Sequence Block
Event

27. Sequence Blocks, Events, and Expectations Mapping
Expectations mapping will be pulled from your school’s curriculum management system, if you have mapped your curriculum expectations
The only mapping Curriculum Administrators must document outside of the curriculum management system is matching School (Program) Expectations to PCRS CI
PCRS
Program
Sequence Block
Event
School CMS
This slide takes the general concepts about ways schools can map expectations in their curricula back to the Curriculum Inventory and stresses that those expectations are done within their curriculum management systems – whatever mapping schools do in their local curriculum management system (even if it is no mapping at all) will be reflected in the Curriculum Inventory without any additional effort . BUT PCRS is done outside of curriculum management systems – someone (Curriculum Dean, Curriculum Committee, Curriculum Staff) must match the school Program Expectations to PCRS once (on paper or using a tool provided by their curriculum management system vendor or developer).

28. Sequence Blocks, Events, and Expectations Mapping
Mapping Program Expectations to PCRS and Event and/ or Sequence Block Expectations to Program Expectations allows for reporting of PCRS to the Event / Sequence Block Level:
Expectations being mapped across curricula
Instructional methods used to teach expectations
Assessment methods used to assess achievement of expectations
Content areas in which expectations are taught
PCRS
Program
Sequence Block
Event
Matching of School Program Expectations to PCRS will be incorporated into any mapping done within the curriculum management system, and that allows the Curriculum Inventory to map to Sequence Block and / or Event Expectations that were mapped to Program Expectations across all US and Canadian medical and osteopathic schools. This will result in exciting new opportunities to research and report on where PCRS competencies are being taught, how they are being taught, how they are being assessed, and the content areas in which they are being taught.

29. CI Verification Report
Institution (Program Information)
All Academic Levels (Dates)
All Sequence Blocks
Expectations / Expectation Mapping (if provided)
All Events
Expectation Mapping (if available)
Instructional Methods
Primary Instructional Methods
Assessments
Events with Assessment Method(s) only

31. CI Verification Report

32. CI Verification Report

33. CI Verification Report

34. CI Verification Report

35. CI Verification Report

36. CI Verification Report

37. CI Verification Report

38. Curriculum Inventory Process

39. Curriculum Inventory Process
Portal
Manage Data (Home Screen)
Download Verification Report
After successful data upload
Reject or Verify Data
Manage Users (Four Roles)
Curriculum Dean (Pre-populated)
Primary CI Admin (Pre-Populated)
CI Admin
CI Viewer
Schools are encouraged to add Vendor with CI Viewer role (work with vendor to create )
Manage Vendors
Vendor must be selected by school in Portal or data will not upload

40. Curriculum Inventory Process
Data
Submit Data
‘Portal Open’ Notification: August 1
System Check for (MB CI Standard, Terminology, Competencies)
Verify Data
Verification Report available: soon after data is successfully submitted (15 – 30 minutes)
Deadline for verifying data: September 30
‘Portal Closed’ Notification: October 1
Create Snapshot of data

41. Curriculum Inventory Process
Reminders and Notifications
Primary CI Admin / Curriculum Dean
Dean
Ramifications
No access to Curriculum Inventory Reports
No documentation for continuous quality improvement and accreditation
Portal
Only available actions:
Download Verification Report
Manage Users
Manage Vendors

42. Upload Process Vendor / Sender Dependent CMS exports XML file
File is uploaded through CI Portal
Web services uploads from CMS to AAMC
XML file is manually created

43. School Portal

44. Curriculum Inventory Process
Please submit and verify data prior to 09/30.
An activity log displays all activity – data upload attempts and status, Verification Report downloads, and data rejections and verifications.

45. Curriculum Inventory Process
09/30/2015.
August 30
August 30.

46. Curriculum Inventory Process
09/30/2015.
August 30
August 30

47. Curriculum Inventory Process
09/30/2015.
August 30
August 30
August 30
August 30
The data and Verification Report will remain available until the upload process for the next academic year begins.

48. CI Submission Errors Two types of submission errors:
Core: upload file is not processed; notification goes to sender; no status update in CI Portal Activity Log
Incorrect or missing AAMC Institution ID
Sender not selected by school in CI Portal
Upload includes data for more than one school
Upload does not pass XML format check
Business Rules: Error message is sent to CI Portal
Program Level Competencies not matched to Physician Competency Reference Set (PCRS)
Instructional and / or Assessment Method that is not on Standardized Vocabulary list

49. AAMC CI Business Rules

50. AAMC CI Business Rules

51. What Data Should be Included in CI Uploads?
This section highlights some of the specifics from the CI Standard Specification and the AAMC Business Rules – it is not an inclusive list of every requirement.

52. Curriculum Inventory Data
Events
Duration in hours and / or minutes
Instructional Methods
Can have multiple IMs but one must be designated Primary
Assessment Methods
Summative vs Formative
Assessment Events
Resources
Expectations (Learning Objectives / Outcome Objectives / Competencies)
Keywords
(School-generated, LCME “Hot Topics,” curriculum content terms for LCME documentation, USMLE Content Outline, MESH, UMLS, etc.)

53. Curriculum Inventory Data
Sequence Blocks
ALL Sequence Blocks: Courses, Clerkships, Electives
Duration in Days AND / OR Begin / End Dates
Clerkships in Days AND Begin / End Dates
Days for single iteration (5 days = 1 week)
Begin / End Dates for Iteration Period
Expectations (Learning Objectives / Outcome Objectives / Competencies)
Nested Sequence Blocks (Optional)
Organ Systems
Integrated Clerkships / Longitudinal Clerkships
Selectives
Integration Blocks (Optional)
Themes
Vertical Integration / Horizontal Integration

54. Issues/Challenges
Creating a crosswalk to map ‘local’ vocabulary to Standardized vocabulary
Implementing the concept of Resources for Instructional and Assessment Methods
Linking course and session objectives to program objectives / competencies
Clerkship data – what needs to be documented; where can the data be found?
Electives data – what needs to be documented; where can the data be found?
Uploading CI data in a manner that will meet needs for continuous quality improvement and accreditation

55. CI Upload Process Biggest Hurdles: Report Start and End Dates
Institution Information
Competencies
Program to PCRS (related)
Hierarchy Levels (broader / narrower)
Academic Levels
Start and End Dates
Sequence Blocks in multiple levels
Unique Curricula
Multiple options for students

56. CI Upload Process Biggest Hurdles: Data accuracy Organizing data
Documenting competencies in XML
CI Verification Report
Assessment Counts
Summative / Formative
Instructional Method Hour Counts
Constantly-changing environment
Communication!

57. Documenting Multiple Iterations
Goal is to document what the majority of students experience
CMS and CI document faculty expectations of how students will progress through the curriculum – NOT individual student progress
Challenges:
Small Groups
Multiple Clerkship Iterations
Multiple Sites
Multiple Tracks
Multiple Campuses

58. Documenting Multiple Iterations
Upload options:
Choose one student / student group for upload
System automatically uploads first or last instance of each iteration
Create ‘Model’ instance of each iteration
Upload all iterations
Issue: Are these iterations distinguishable as multiple iterations?

59. Documenting Multiple Iterations
Aggregate report options:
Quarantine outliers
Calculate averages per institution for all iterations of Sequence Blocks
Change explanations or add footnotes for numbers of schools

60. Documenting Clerkships:
Goal is to document as many hours as possible
Clerkship weeks x 50 hours
Didactics
Include X-Ray Conference, M&M, Tumor Board, Grand Rounds
Individual activities
Scheduled clinical activities
Clinics, surgeries, mentors, attending rounds, resident rounds
Unscheduled clinical activities
Divide ‘undocumented hours’ using ambulatory / inpatient percentage

61. Documenting Clerkships:
Metadata
Objectives and Keywords
Course syllabus
Didactic / Conference Slides
Evaluation system data
Faculty / Student documentation
Competencies
Keywords

62. Best Practices for Curriculum Documentation
Centralized
Review syllabi and educational materials
Interview faculty
Distribute worksheets to faculty
Provide an online tool for faculty
Hire medical students
Search software
Natural language processing

63. Best Practices for Curriculum Documentation
Distributed
Faculty document as educational materials are uploaded to LMS
Faculty provide course coordinators with mapping terminology
Course Directors collect and document all mapping terminology for their courses

64. Preparing for 2016-2017 Staging (for testing uploads) available now!
Web Services (requires use of SOAPUI)
Staging Portal (requires staging-specific credentials – send request for access to
Webinars/Virtual Focus Groups:
Three-part Developer Series
Curriculum Deans and Administrators
Curriculum Mapping
Documenting Challenging Content

65. Documenting Resources:
Use of resources is optional, but encouraged
Documents use of Standardized Patients, Virtual Patients, etc.
Beginning with , if Resources are uploaded, Standardized Terminology UIDs will be required

66. What’s next? Two major initiatives: Data validation tool
Keyword synonym check

67. CI Data Analysis Issue 1: copying one course over to all other courses
Issue 2: keywords that just have letters or punctuation
Issue 3: synonyms
Issue 4: no titles for competency objects

68. CI for CQI and Accreditation Task Force
Charge:
Review the types of data in the Curriculum Inventory
Analyze how well that CI data and the way it is formatted responds to the requirements of the DCI
Develop best practices to assist schools as they upload to the CI in preparation for completing a DCI

69. For more information www.aamc.org/cir Reports Webinars and training
Information about Curriculum Management System vendors
Tools for Developers
Curriculum Inventory School Portal

70. Questions?