1. Preparing for the Curriculum Inventory Terri Cameron, MA Director, Curriculum Programs AAMC

2. We will cover: Quick Introduction to Curriculum Inventory MedBiquitous Curriculum Inventory Standard AAMC Business Rules for Curriculum Inventory data What curriculum management system data should be included in the XML export to the Curriculum Inventory Vendor / Sender Onboarding Process Data Upload Process Data Verification Process

3. What is the Curriculum Inventory An international repository of curriculum information that can be used to report on what is being taught in medical schools, how content is being taught, how content is being assessed, the resources used to teach and assess, and the competencies taught in conjunction with the content.

4. Who Participates in the CI? 2015: 135 Schools 127 USMD 5 CAMD 3 DO (Pilot) 2014: 120 Schools 115 USMD 5 CAMD Inaugural Year (2013): 90 Schools 85 USMD 5 CAMD

5. Why participate in the CI? The Curriculum Inventory is the culmination of a long history of the AAMC collecting information about medical education content, structure, and pedagogy to support: Medical Education Research (medical school faculty and external researchers) Continuous Quality Improvement / Benchmarking Support of Curriculum Committees Reporting on new trends Responding to legislative inquiries or providing data when important issues are being discussed Respond to media inquiries The Curriculum Inventory is only effective when the data represents most, if not all, medical schools participate. Only participating schools can request detailed Curriculum Inventory Reports.

6. Continuous Quality Improvement and Accreditation Curriculum Inventory Reports MedBiquitous Data Exchange Standard Curriculum data mapped to a competency hierarchy (competencies to course objectives to session objectives) are uploaded to the Curriculum Inventory (CI) from school systems using the MedBiquitous CI data exchange standard (www.medbiq.org).www.medbiq.org Standardized Vocabulary for Instructional Methods, Assessment Methods, and Resources Institutionally- developed competencies will be matched to Physician Competency Reference Set (PCRS) Vendor and School – developed Curriculum Management Systems Curriculum Inventory Part II

7. Questions? Please use Q & A option to submit questions.

8. Terminology

9. Previous Academic Year (LCME model – if current year is 2013-2014, previous academic year is 2012-2013) Academic Level / Phase (formerly called ‘year of curriculum’) Sequence Block /Nested Sequence Block (Course, Module, Unit, Block, Clerkship) Integration Block (Theme, Vertical / Horizontal Integration) Event (Instructional or Assessment Session) (e.g., Lecture, Lab, Discussion, Assessment) Expectations / Competency Framework (Competency / Objective / Milestone / EPA) MedBiquitous (www.medbiq.org)www.medbiq.org Part II

10. Standardized Vocabulary  Use local terms for institutional data entry and reporting; match to standardized vocabulary to upload to CIR for aggregate reporting  Instructional Methods  Assessment Methods  Resources  UIDs available at https://www.aamc.org/download/348854/data/cibusinessrules.pdf https://www.aamc.org/download/348854/data/cibusinessrules.pdf  Detailed descriptions available at www.mededportal.org/icollaborative/resource/498

11. Curriculum Inventory Standardized Vocabulary Update Renamed Renamed: Exam – Institutionally Developed, Laboratory Practical

12. Matching Local Terminology to CI Standardized Terminology Local Terminology: Instructional Methods CI Standardized Terminology Clinical SkillsClinical Experience - Outpatient Procedure TrainingClinical Experience - Inpatient Resource SessionLecture Standardized PatientSimulation (with Resource: Standardized Patient) Wrap-Up SessionDiscussion, Small Group The MedBiquitous Curriculum Inventory Standardized Terminology Subcommittee reviews the terminology each year, based on literature and terminology schools report as difficult to match (challenges) Submit matching ‘challenges’ to medaps@aamc.org or post comments in iCollaborativemedaps@aamc.org

13. Physician Competency Reference Set (PCRS)  Compared/mapped healthcare profession competency sets to create a set of ‘core’ competencies for aggregate reporting in MedAPS and MedEdPORTAL  ACGME (including RRCs)  CanMEDS  Scottish Doctor  Tomorrow’s Doctors  Healthcare Professions  Specialty Objectives (ABMS, Clerkships)  Process and complete list of competencies published in August 2013 Academic Medicine  URI available at https://www.aamc.org/initiatives/cir/about/348808/aboutpcrs.htmlhttps://www.aamc.org/initiatives/cir/about/348808/aboutpcrs.html

15. Physician Competency Reference Set (PCRS) Partial URI Physician Competency Reference Set (AAMC 2013) aamc-pcrs-comp-c0100 PATIENT CARE: Provide patient-centered care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health aamc-pcrs-comp-c0101 Perform all medical, diagnostic, and surgical procedures considered essential for the area of practice aamc-pcrs-comp-c0102 Gather essential and accurate information about patients and their condition through history-taking, physical examination, and the use of laboratory data, imaging, and other tests aamc-pcrs-comp-c0103 Organize and prioritize responsibilities to provide care that is safe, effective, and efficient aamc-pcrs-comp-c0104 Interpret laboratory data, imaging studies, and other tests required for the area of practice aamc-pcrs-comp-c0105Make informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to- date scientific evidence, and clinical judgment

16. Matching School Expectations to PCRS Program-Level (School) Expectations must be mapped to PCRS competencies May NOT be mapped to PCRS domains ‘Other’ provided for instances where there is not a direct match between expectations Event- and Sequence Block-Level Expectations may NOT be mapped to PCRS Event and Sequence Block Expectations may be mapped to each other or to Program Expectations or not at all This mapping is pulled from your curriculum management system PCRSProgram Sequence Block Event

17. Matching Program Objectives to PCRS (Crosswalk) PCRSProgram Objectives 2. Knowledge for Practice: Demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social-behavioral sciences, as well as the application of this knowledge to patient care. 2.1 Demonstrate an investigatory and analytic approach to clinical situations 2.2 Apply established and emerging bio-physical scientific principles fundamental to health care for patients and populations 2.3 Apply established and emerging principles of clinical sciences to diagnostic and therapeutic decision- making, clinical problem-solving, and other aspects of evidence-based health care 2.4 Apply principles of epidemiological sciences to the identification of health problems, risk factors, treatment strategies, resources, and disease prevention/health promotion efforts for patients and populations 2.5 Apply principles of social-behavioral sciences to provision of patient care, including assessment of the impact of psychosocial and cultural influences on health, disease, care-seeking, care compliance, and barriers to and attitudes toward care 2.6 Contribute to the creation, dissemination, application, and translation of new health care knowledge and practices 2.99 Other knowledge for practice 2.Demonstrate a broad working knowledge of the fundamental science, principles, and processes basic to the practice of medicine. 2.1 Understand the clinical relevance of scientific inquiry. 2.2 Apply this knowledge in a judicious and consistent manner to prevent common health problems and achieve effective and safe patient care. 2.3 Demonstrate the ability to evaluate emerging knowledge and research as it applies to diagnosis, treatment and the prevention of disease. 2.4 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of individuals. 2.5 Utilize state of the art information technology and tools to retrieve, manage and use biomedical information in the care of populations. 2.6 Understand the indications, contraindications, and potential complications of common clinical procedures. 2.7 Perform the basic clinical procedures expected of a new PGY-1.

18. Competencies vs Objectives Objectives Competencies Expectations Milestones EPAs Competencies, Learning Objectives, Milestones, and Entrustable Professional Activities (EPAs) are all separate concepts with explicit definitions and purposes. For the Curriculum Inventory, they are all “Expectations”. The category element can be used to distinguish whether an Expectation is a learning objective, competency, etc., but there are no requirements to use that option at this time.

19. Questions? Please use Q & A option to submit questions.

20. The Curriculum Inventory Standard

21. ® Mission: To develop and promote technology standards for the health professions that advance lifelong learning, continuous improvement, and better patient outcomes. Not-for-profit, member-driven, standards development organization

22. Building the Foundation: The MedBiquitous Architecture (www.medbiq.org) * American National Standards † Candidate standard

23. MedBiquitous Curriculum Inventory Working Group Susan Albright and Hugh Stoddard, co-chairs Launched in December 2010 Charter:  “to develop XML standards for the exchange of curriculum data for benchmarking and educational research.” (Not just for AAMC Curriculum Inventory) Reasons for Undertaking Work: Enable local curriculum management systems to upload data to a central Curriculum Inventory Identify curricular trends Support curriculum benchmarking and reform

24. A E B Curriculum Inventory: Events A curriculum is made up of many education and assessment events Events have metadata, descriptive information about the events F C G DH I J K L

25. Curriculum Inventory: Expectations Curricula may reference objectives, competencies, or learning outcomes 1 2 4 5 7 8 1011 3 6 9 12

26. Curriculum Inventory: Expectations and Events An event may be intended to address one or more expectation (learning objectives, competencies, etc.) An expectation may be associated with many events G K 1 2 6 2

27. Curriculum Inventory: Sequence Blocks Events can be organized into blocks of instruction, called Sequence Blocks (think course/module/year/phase/clerk ship etc) A recurring event may be associated with more than one Sequence Block Sequence blocks may be associated with expectations (competencies, objectives, etc.) Sequence blocks have metadata, too A E B F C G DH I J K L A 1 2 7

28. Curriculum Inventory: Integration Blocks Curricular themes, or Integration Blocks, may apply to many sequence blocks, events, or expectations and describe how the curriculum is semantically integrated An Integration Block may link to many events, blocks or objects A E B F C G DH I J K L A M E O N C G SH I J P R Q 1 3 2

29. Curriculum Inventory: Sequence Sequence blocks can be put together to form the main structure of the curriculum (a sequence of courses, modules, phases, clerkships) A E B F C G DH I J K L A M E O N C G SH I J P R Q 1 2 3

30. Includes Competency objects and frameworks Curriculum Inventory: Expectations

31. Instructional and / or Assessment methods Resources (virtual patient, etc.) Keywords Objectives (expectations) Curriculum Inventory:

33. Questions? Please use Q & A option to submit questions.

34. AAMC Business Rules for the Curriculum Inventory

35. AAMC Business Rules Are designed for specific use of the MedBiquitous Curriculum Inventory Standard for use in the AAMC Curriculum Inventory Are not designed to replicate the MedBiquitous Curriculum Inventory Standard or the MedBiquitous Curriculum Inventory Implementation Guidelines (www.medbiq.org)www.medbiq.org Should be used as a major part of the CI Developers Toolkit, but not in isolation

36. AAMC Business Rules Core Rules (CR) Core rules must be met before additional rules for the competency framework and the curriculum inventory structure are applied. If it is detected that a submission does not meet a core rule: a) processing of the submission discontinues immediately and b) an error notification is generated that describes the invalid core rule.

37. AAMC Business Rules Core Rules (CR) CR01: Approved Sender Senders must be certified via AAMC’s on- boarding process before the web service will allow them to send submissions. CR02: Selected Sender – Schools must select their Sender via the School CI Portal ‘Manage Vendor’ page. o Vendor o School

38. AAMC Business Rules Core Rules (CR) CR03: One Inventory for One School A submission, or XML file sent to the web service, must include data for only one curriculum inventory for one school. CR04: Institution ID The must match an AAMC institution ID for a school. AAMC EIS Code

39. AAMC Business Rules Core Rules (CR) CR05: Submission File Size The file size cannot exceed 19mg An error will be generated if data in any field exceeds the record length Zip files accepted CR06: Data Dictionary Each field in the XML has a specific length limit, which is documented in the AAMC Business Rules. An error will be generated if data in any field exceeds its maximum allowable length

40. AAMC Business Rules Competency Framework and Competency Object Rules (CF) These rules are evaluated only once all core rules have been met. These rules and their application are covered in CI Developers Workshop Part II.

41. AAMC Business Rules Curriculum Structure Rules (CS) These rules are evaluated only once all core rules have been met.

42. AAMC Business Rules Curriculum Structure Rules (CS) CS01: Unique Report ID The sub-element must be unique for each of a school’s submissions. If a Curriculum Inventory Upload is rejected, the next uploaded file must not have the same ReportID. The sub-element must be alphanumeric The Report ID must not include dashes, underscores or other special characters

43. AAMC Business Rules Curriculum Structure Rules (CS) CS02: Previous Academic Year Submissions are checked to ensure they contain information for only the previous academic year (AY). Schools must submit data with and between July (of the previous year) and June (of the current year). Note: Dates at the sequence block or event level are not checked to ensure they are within the previous academic year. Dates specified at these levels that fall outside of the date range for a previous academic year will not cause an error or rejection.

44. AAMC Business Rules Curriculum Structure Rules (CS) CS03: Number of Academic Levels The number of academic levels, or phases, are defined by. Each academic level is defined as a within the tag. CS05: Academic Level Numbers sub-elements of the tag must have sequential numbers (i.e., the number attribute) starting with 1.

45. AAMC Business Rules Curriculum Structure Rules (CS) CS04: Academic Levels and Sequence Blocks A that is not defined within the tag cannot be referenced by a sequence block. CS06: Academic Levels and Sequence Blocks Each academic level must be referenced by at least one sequence block.

46. AAMC Business Rules Sequence Blocks CS07: tags must have unique identifiers (i.e., the id attribute). CS08: tags must be associated to a defined academic level using the required sub- element. CS09: The sub-element of all tags with a ClerkshipModel attribute must specify and. Document start and end dates for entire iteration period (e.g., 07/01/12 – 06/30/13). CS16: If it is provided, the duration of a sequence block must be specified as a number of days. For clerkships, 5 days = 1 week

47. AAMC Business Rules Curriculum Structure Rules (CS) CS10: Nested Sequence Blocks When nesting sequence blocks, must not result in circular references. See “Note about Hierarchical Conflict” section (pp. 33-36) of the Competency Framework Specification, version 1.0.

48. AAMC Business Rules Curriculum Structure Rules (CS) Events CS11: tags must have unique identifiers (i.e., the id attribute). CS12: Within tags, the and tags must reference a unique ID from the Curriculum Inventory Standardized Vocabulary. CS13: At a minimum, each must reference at least one or. CS14: If one or more tags are provided, one and only one must be denoted as primary. CS15: All tags must be referenced by at least one tag within a.

49. Questions? Please use Q & A option to submit questions.

50. Curriculum Inventory Process

51. Portal Manage Data (Home Screen) Download Verification Report After successful data upload Reject or Verify Data Manage Users Schools will be encouraged add Vendor with CI Viewer role Manage Vendors Vendor must be selected by school in Portal or data will not upload

52. Curriculum Inventory Process Data Submit Data ‘Portal Open’ Notification sent 08/01 System Check for (MB CI Standard, Terminology, Competencies) Verify Data Verification Report available as soon as data is successfully submitted Deadline for verifying data is 09/30 ‘Portal Closed’ Notification sent 10/01 Create Snapshot of data Reports process begins immediately

53. Curriculum Inventory Process Reminders and Notifications Curriculum Dean / Primary CI Admin Dean Ramifications MedAPS Reports LCME Portal Only available actions: Download Verification Report Manage Users Manage Vendors

54. Upload Process Vendor / Sender Dependent CMS stores data on School server CMS exports XML file File is manually uploaded by School (or Vendor if Vendor has CI Primary Admin role) CMS stores data on Vendor server Web services upload from Sender to AAMC

55. Questions? Please use Q & A option to submit questions.

56. School Portal

57. Curriculum Inventory Process Portal Manage Data (Home Screen) Download Verification Report After successful data upload Reject or Verify Data Manage Users (Four Roles) Curriculum Dean (Pre-populated) Primary CI Admin (Pre-Populated) CI Admin CI Viewer Schools are encouraged to add Vendor with CI Viewer role (work with vendor to create e-mail) Manage Vendors Vendor must be selected by school in Portal or data will not upload

58. Web Services

59. Curriculum Inventory Process An activity log displays all activity – data upload attempts and status, Verification Report downloads, and data rejections and verifications.

60. Curriculum Inventory Process

62. The data and Verification Report will remain available until the upload process for the next academic year begins

63. CI Verification Report Institution (Program Information) All Academic Levels (Dates) All Sequence Blocks Expectations / Expectation Mapping (if provided) All Events Expectation Mapping (if available) Instructional Methods Primary Instructional Methods Assessments Events with Assessment Method(s) only

65. CI Verification Report

72. Questions? Please use Q & A option to submit questions.

73. Pearls A Sequence Block Week is 5 Days Clerkships are Sequence Blocks with Clerkship Model (Rotation or Longitudinal) rotation Check date / duration formats 2011-07-01 2010-07-01 2011-06-30 PT2H PT90M PT1H30M P13D P12M

74. Pearls Events must have Instructional Method OR Assessment Method(s) OR Both Use Instructional Method and Assessment Method UIDs!!!! IM013 AM012 Can have multiple Instructional Methods but one must be Primary Can have multiple Assessment Methods – no Primary Must have Duration Formative / Summative Assessment Events No Instructional Methods

77. Pearls Country Code: 3-character code (USA, CAN) USA

78. Pearls Program Code: 2-character code (MD, DO, PA) M.D. 5678 Parallel Tracks Denote same cohort of students in separate tracks (should not be used to designate concurrent Academic Levels for multiple cohorts of students)

79. Pearls Special Characters: When data is copied from a website and pasted into Curriculum Management Systems, the special characters that make up the HTML can cause problems either in the upload process OR in reporting. In addition, these special characters sometimes cause unexpected truncation because they fill the character limit for the field. Solution: when possible, use the paste function ‘text only’ when pasting text into a CMS

80. Pearls Extra spaces: /CurriculumInventory/Events/Eve nt[@id = 'E395'] This issue often makes it through the error checking and results in a successful upload, but the Verification Report is not generated

81. Issues Found in Data Analysis Issue 1: copying one course over to all other courses Issue 2: keywords that just have letters or punctuation Issue 3: synonyms Issue 4: no titles for competency objects Issue 5: Uploading only one event for all sequence blocks Issue 6: Uploading only one academic level

82. 12345 NoSuch University School of Medicine 987 720 Main Street Baltimore MD 21205 US M.D. 5678 NoSuch School of Medicine Curriculum 2010-2011 2011-07-01 2010-07-01 2011-06-30 en-us The NoSuch curriculum reframes the context of health and illness to encourage students to explore a larger, integrated system. http://www.nosuchmedicine.org/crc/

83. Questions? Please use Q & A option to submit questions.

84. 2015-2016 Schedule Staging available now School Portal Opens: August 1 School Portal Closes: September 30 Status e-mail sent to CI Primary Admin, Curriculum Dean, and School Dean: September 30 Data clean-up/reporting starts: November 1

85. Resources Curriculum Inventory Implementers’ Listserv: http://medbiq.org/mailman/listinfo/ciimplementer_medbiq.org http://medbiq.org/mailman/listinfo/ciimplementer_medbiq.org Competency Framework Specs and Schemas: http://www.medbiq.org/working_groups/competencies/index.html http://www.medbiq.org/working_groups/competencies/index.html AAMC Curriculum Inventory Developer Resources: https://www.aamc.org/initiatives/cir/about/349624/fordevelopers.html https://www.aamc.org/initiatives/cir/about/349624/fordevelopers.html Physician Competency Reference Set URIs: https://www.aamc.org/external/348732?url=https://www.mededportal.org/ic ollaborative/resource/793 https://www.aamc.org/external/348732?url=https://www.mededportal.org/ic ollaborative/resource/793 MedBiquitous: www.medbiq.org www.medbiq.org Curriculum Inventory Working Group: http://www.medbiq.org/curriculum_inventory Competencies Working Group: http://www.medbiq.org/working_groups/competencies/index.html

88. Questions? List of upcoming Curriculum Inventory Webinars and materials from today’s session: www.aamc.org/cir