1. Clinical Programs and Competence Evaluation and Outcomes Keith Watson, D.O. Associate Dean for Graduate Medical Education Centers for Osteopathic Research and Education

2. Basic Science Clinical Science D.O. B.E. PGY2 Osteopathic Principles and Practice OOO O Outcomes Measures Osteopathic Continuum Curriculum Medical School Residency

3. Basic Science Clinical Science D.O. B.E. PGY2 OO O Osteopathic Continuum Curriculum Medical School FM Residency I II III OSCEOSCE?OSCE In-service ? COMLEX Osteopathic Principles and Practice Cert. Exam

4. Resident Evaluation External validation (examples) In-service examinations Certification pass rates for graduates (first time takers) Internal validation (examples) Continuous Progress Indicator exam (CPI) Objective Structured Clinical Examination (OSCE) [Objective Structured Assessment of Technical Skill (OSATS)] Performance Assessments of rotations.

5. General Principles –Evaluation Standards Language must reflect evaluations required for minimal approval Generalized statements are often not helpful Regular Self Study Review and updated Goals Statements should be required for at least: Growth Improvement Outcomes Consortium Goals for OPTI Sponsor must be integrated into Program Self Study and Objectives

6. Program Evaluation External validation (examples) Group scores on In Service Examination Collective pass rates on Board Examinations Peer reviewed assessments every five years Internal validation (examples) Self Study Involving PD, faculty, and Institution Identification of problems Identification of Markers of Excellence Goal Setting Ongoing regular review of every stated goal and present outcome Etc.

7. Program Evaluation-How well does the program meet its stated goals? Basic Standards = minimums Self Study and Declared Plans (Targets) for: Growth goals (examples) increase recruitment/retention by _% by ___. add ___ number of compensated faculty by ____. Improvement goals (examples) institute quality assurance thresholds for clinic charts by ___ adjust curricular emphasis to improve resident performance on In Service Examination by ___% Outcome goals (examples) Achieve 100% graduate pass rate (first attempt) on Board Certification Exam All residents will demonstrate a high proficiency rating in each of the seven competencies prior to graduation

8. Program Evaluation – OPTI Considerations Standards should address: Adherence to OPTI sponsor policies and procedures regarding evaluation expectations and outcomes Coordination of program outcomes with general OPTI sponsor goals and outcomes Shared data resources with OPTI sponsor and the restrictions/conditions of use

9. Information Silos Create Barriers Programs and OPTIs must access and share data: NBOME Specialty Colleges Certification Boards COPT Etc

10. Resident Evaluation External validation (examples) In-service examinations Certification pass rates for graduates (first time takers) Internal validation (examples) Continuous Progress Indicator exam (CPI) Objective Structured Clinical Examination (OSCE) [Objective Structured Assessment of Technical Skill (OSATS)] Performance Assessments of rotations.

11. Evaluation Goals Measure objectives in more than one domain: Affective (attitudinal) Psychomotor (skills) Cognitive (knowledge) Choose evaluation tool which best fits the domain and performance area: No one tool is the best fit for all parameters

12. Knows Knows How Shows How Does Assessment Model

13. Guide for Selecting a Testing Method (A=Most efficient/B=Secondary utility) Knows How Shows How Does Written tests – essay, MCQ AA Written exercises – application test BA Nonpatient exercises- computer based, practical exam BA Patient exercises- OSCE, SPs BA Observations- direct, MR review, oral exam, peer BA

14. Conclusions More than one type of evaluation should be used for overall assessment of a resident. PA may be summative; a formative PA should be used for interval progress discussion. SP experience essential For experience toward licensure/certification Best method for demonstrating skills learned Testing domains should be linked to articulated learning objective domains. Entire process should be congruent with declared outcome objectives for the residency (evidence based).

15. Osteopathic General Competencies for Residents Osteopathic Manipulative Therapy Patient care Medical knowledge Practice-based learning and improvement Interpersonal and communication skills Professionalism Systems-based practice

16. Testing the General Competencies ? Evaluation Method Patient Care (and OMT) Medical Knowledge Practice based learning Interpersonal skills Profes- sionalism Systems Based Practice Written Tests xx? Direct Obser- vation xx?xxx? OSCE xxxxx xxx Perfor- mance Appraisal x?xxxxx Chart Reviews x?x??xx Oral Exams xxxx

17. Writing and Testing Objectives Domains Competency Affective (attitudinal) Psychomotor (skills) Cognitive (knowledge) Patient Care Medical Knowledge Practice based learning Interpersonal skills Professionalism Systems based practice Testing Heirarchy Knows, Knows how, Shows How, Does? Knows, Knows how, Shows how, Does?

18. Basic Science Clinical Science D.O. B.E. PGY2 OO O Osteopathic Continuum Curriculum Medical School FM Residency I II III OSCEOSCE?OSCE In-service ? COMLEX Osteopathic Principles and Practice Cert. Exam

19. Summary Standards should identify minimum outcomes and evaluation processes Standards should require programs to demonstrate self study activity and declare outcome goals Standards should encourage (identify?) outcome markers of excellence which rise above the minimum expectations