1. MBBS, MPH, MCPS, MRCGP (UK), FRIPH (UK), FHAE (UK) TRAINEE EVALUATION METHOD Ass. Prof. Dr. Abdul Sattar KHAN Family & Community Medicine Department College of Medicine King Faisal University

2. Objectives At the end of the session, participants are expected to know: What is an assessment process? Who should assess the students? Why assess the students? What should be assessed? How should the students be assessed? When should the students be assessed? Where should the students be assessed? 10/13/2015 ASK 2

3. What is Learning? A curricular definition Learning - acquiring knowledge, skills, attitudes, and values and experiences. A cognitive definition Learning - a process of formulating new and more complex understandings of the world Learning - revising and refining mental constructs, i.e., the understandings that guide how we think, speak and behave http://dakota.fmpdata.net/PsychAI/PrintFiles/DefLrng.pdf 10/13/2015 ASK 3

4. How do we know that learning is done? 10/13/2015 ASK 4

5. Key Relationship in Learning Process Teaching Learning Assessment 10/13/2015 ASK 5

6. What is an assessment process? 10/13/2015 ASK 6 The Oxford Dictionary 1 defines assessment as ‘the action of assessing’. In the context of medical education, assessment could be defined as determining the competence of the product, that is, the Health Professional. It has three pillars 2 : 1.Cognitive, 2.Affective 3.psychomotor 2. Newble D. Assessment. In: Jolly B, Rees L. (Editors) Medical education in the millennium, Oxford: Oxford University Press; 1998.p.131–42. 1. Oxford Dictionary of English, 2nd Edition, Oxford University Press; 2012

7. Who should assess the students?  Medical schools  International accrediting body  National accrediting body  Professional bodies  Licensure bodies  Department  Individual teacher  patients / community  Students themselves 10/13/2015 ASK 7

8. 8 Why assess the students?  “fit for purpose”  Ranking of student  Measurement of improvement in a student  To diagnose student difficulties  Evaluation of teaching method  Motivating student to study  Provision of feedback for the teacher 10/13/2015 ASK

9. What should be assessed? Competencies / Outcomes CanMeds roles Medical expert Communicator Collaborator Manager Health advocate Scholar Professional ACGME (Accreditation council for Graduate Medical Education) competencies Medical knowledge Patient care Practice-based learning & improvement Interpersonal and communication skills Professionalism Systems-based practice www.royalcollege.ca/common/documents/ca nmeds/ http://www.acgme.org/acgmeweb/ 10/13/2015 ASK 9

10. Miller’s pyramid of competence Knows Shows how Knows how Does Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. 10/13/2015 ASK 10

11. How should the students be assessed? 10/13/2015 ASK 11  Essay Type Questions  Long & Short  Modified Essay  Objective type Questions  Multiple Choice Question MCQ  Supply-item  True-False item  Extending matching item

12. Assessing know Knows Shows how Knows how Does Knows Essay type, MEQ, MCQ etc 10/13/2015 ASK 12

13. Relative characteristics of different types of examination ApplicationsEssayShort answer MCQExtended Matching MEQ Application of knowledge ExcellentGoodPoorGoodExcellent AssessmentExcellentGoodPoorFairGood Coverage of topicPoorGoodExcellent Reliability of scorePoor to Fair GoodExcellent Ease of scoringPoorFairExcellent Preparation timeMin to Mod ModLargeMod Total costLargeModLow Mod CheatingDifficult Easy Difficult 10/13/2015 ASK 13 University of Dundee, 2004 How should the students be assessed?

14. 10/13/2015 ASK 14  Assessment of skills  Patient Management Problem PMP.  Observational Assessment, OSCE, OSPE. MiniCEX, DOPS etc. A new approach: 1.Response Format ( Open-ended & MCQ) 2.Stimulus Format (Context-free & Context-rich – EMQ & Key-feature) 3.Hybrid Format (Script Concordance Test- SCT)

15. Assessing knowing how Knows Shows how Knows how Does Knows Knows how Written complex simulations (PMPs) 10/13/2015 ASK 15

16. Assessing showing how Knows Shows how Knows how Does Knows howShows how Performance assessment in vitro (OSCE) 10/13/2015 ASK 16

17. Assessing does Knows Shows how Knows how Does Shows how Does Performance assessment in vivo by judging work samples (Mini-CEX, CBD, MSF, DOPS, Portfolio) 10/13/2015 ASK 17

18. Tomorrow’s Doctors Assessments will be fit for purpose – that is: Valid Reliable Generalizable Feasible Fair 10/13/2015 ASK 18 http://www.gmc- uk.org/education/undergraduate/tomorrows _doctors.asp

19. How be sure the process is appropriate? anatomy physiology int medicine surgery psychology item pool review committee test administration item analyses student comments Info to usersitem bank Pre-test review Post-test review 10/13/2015 ASK 19 Maastricht Review Process

20. Reliabilities across methods Testing Time in Hours 1 2 4 8 MCQ 1 0.62 0.76 0.93 Case- Based Short Essay 2 0.68 0.73 0.84 0.82 PMP 1 0.36 0.53 0.69 0.82 Oral Exam 3 0.50 0.69 0.82 0.90 Long Case 4 0.60 0.75 0.86 0.90 OSCE 5 0.47 0.64 0.78 0.88 Practice Video Assess- ment 7 0.62 0.76 0.93 1 Norcini et al., 1985 2 Stalenhoef-Halling et al., 1990 3 Swanson, 1987 4 Wass et al., 2001 5 Petrusa, 2002 6 Norcini et al., 1999 In- cognito SPs 8 0.61 0.76 0.92 0.93 Mini CEX 6 0.73 0.84 0.92 0.96 7 Ram et al., 1999 8 Gorter, 2002 10/13/2015 ASK 20 Cees van der Vleuten, 2010

21. Factors influencing reliability  Environmental errors  Processing errors  Generalization errors  Bias errors a. Weighting b. Rater prejudice c. Halo effect d. Leniency and stringency 10/13/2015 ASK 21

22. Factors influencing validity  Unclear instructions  Complicated vocabulary  Too easy/difficult items  Unintentional clues  Not matching with outcome  Inadequate time  Too few items  Unsuitable item arrangement  Detectable answer pattern 10/13/2015 ASK 22

23. Measuring the unmeasurable Knows Shows how Knows how Does “Domain independent” skills “Domain specific” skills Assessment (mostly in vivo) heavily relying on expert judgment and qualitative information 10/13/2015 ASK 23 Cees van der Vleuten, 2010

24. Measuring the unmeasurable Self assessment Peer assessment Co-assessment (combined self, peer, teacher assessment) Multisource feedback Log book/diary Learning process simulations/evaluations Product-evaluations Portfolio assessment 10/13/2015 ASK 24 Cees van der Vleuten, 2010

25. When should the students be assessed? At the beginning of the course During the course (Formative - Diagnostic) At the end of the course (Summative - Certifying) 10/13/2015 ASK 25

26. Where should the students be assessed? Examination hall Class room Hospital wards Out patients department (OPD) Work Places Community 10/13/2015 ASK 26

27. The Myths of testing Testing motivates by threating If I threaten you will fail and then you will try harder Maximize anxiety to maximize learning Testing helps teachers make important instructional decisions Students are not assessment users 10/13/2015 ASK 27 (Stiggins, 2004)

28. The Myths of testing Important assessment decisions can be made once a year Investment of time, effort, and money into large-scale testing supports this belief Learning how to assess is not as important as learning how to teach Teachers teach and testing professionals test 10/13/2015 ASK 28 (Stiggins, 2004)

29. Take Home Message When assessing students it is important to familiar with maximum methods. So, use as many as possible… 10/13/2015 ASK 29