Presentation is loading. Please wait.

Presentation is loading. Please wait.

programmatic assessment for learning

Similar presentations


Presentation on theme: "programmatic assessment for learning"— Presentation transcript:

1 programmatic assessment for learning
an example of medical education design

2 assignment Build an assessment programme for a workplace-based learning curriculum. GP training practice, assignments plus day-release education 1 year, two semesters, 4 terms supervisor, medical educator, on-line platform

3 overview a bit of history programmatic assessment
the assessment programme

4 old model of medical competence
knowledge skills problem solving attitudes TEST TEST TEST TEST TEST TEST TEST TEST

5 history the quest for the best test - oral versus written
- open versus closed items - computer versus paper-and-pencil - knowledge versus insight - norm referenced versus criterion referenced ....and many more

6 typical approach to assessment
pass competent test fail incompetent

7 but…. competent incompetent pass fail

8 major problems no extrinsic standards for criterion validity exist competence is much more complex than expected what does that mean for quality of a test? assessment is not non-obtrusive, so succeptible to many biases and unwanted effects

9 quality elements of assessment
R V E C A R = reliability V = validity E = educational impact C = costs A = acceptance Van der Vleuten CPM. The assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Science Education 1996;1(1):41-67.

10 reliability test’ test Penny 85% Penny Leonard Amy Howard 85% 73% 59%
51% Leonard 73% cut-off Amy 59% Howard 51%

11 validity: Kane’s view practice observation for clinical reasoning
expertise, observation standards, scales, forms observation reliability, reproducibility saturation, expertise observed score relation with multiple choice key-feature, EMQ universe score think aloud, CRT, SCT target domain construct

12 educational impact content format scheduling regulatory structure
curriculum assessment content format teachers students scheduling regulatory structure

13 educational impact Cilliers, F. J., Schuwirth, L. W. T., Herman, N., Adendorff, H., & van der Vleuten, C. P. M. (2012). A model of the pre-assessment learning effects of summative assessment in medical education. Advances in Health Sciences Education, 17(1),

14 quality elements of assessment
Rw Vw Ew Cw Aw R = reliability V = validity E = educational impact C = costs A = acceptance W = weight Van der Vleuten CPM. The assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Science Education 1996;1(1):41-67.

15 But…. testing requires some strange assumptions

16 underlying concepts traits are stable and generic characteristics

17 underlying concepts: stable trait
1 2 3 4 T 1’ 2’ 3’ 4’ A B C 1 .5 4 2 PT A 1 1 1 1 4 B .5 .5 .5 .5 2 C

18 underlying concepts: stable trait
1 2 3 4 T 1’ 2’ 3’ 4’ A B C 1 .1 .5 2 1.5 PT A .5 .5 1 B 1 .5 1 2.5 C 1 1 .5 1 3.5

19 underlying concepts traits are stable and generic characteristics
individual items in themselves are meaningless

20 underlying concepts: meaningless items
Ms. Smit is 72 years old. She has angina pectoris. Several times her blood pressure is taken and found to be 170/100 mmHg. Which antihypertensive drug is most indicated for her?? a captopril. b chloorthalidon. c metoprolol.

21 underlying concepts: meaningless items
Mr. Johnson, 35 years old, consults his GP with complaints of chest pain. Without further information about Mr. Johnson the most likely origin of his chest pain is: a the chest wall; b the lungs; c the myocardium; d the esophagus.

22 underlying concepts: meaningless items
resuscitation ‘station’ in a skills test

23 underlying concepts: meaningless items
communication ‘station’ in a skills test

24 underlying concepts traits are stable and generic characteristics
individual items in themselves are meaningless sum scores determine what the test measures statistics are based on elimination of information

25 underlying concepts: reductionism
1 2 3 4 5 6 a c b e answer b c a e key 0% 100% 1 50% failed

26 underlying concepts traits are stable and generic characteristics
individual items in themselves are meaningless sum scores determine what the test measures statistics are based on elimination of information one single best instrument for each trait

27 old model of medical competence
knowledge skils problem solving attitudes TEST TEST TEST TEST TEST TEST TEST TEST

28 competencies competencies are simple or more complex
tasks a successful candidate must be able to handle, and during which s/he uses at the right time the correct and relevant knowledge, skills, attitudes and meta-cognitions to manage the situation successfully.

29 competency domains or roles
National Dutch blue print: 1 medical expert scientist worker in the health care system person

30 overview a bit of history programmatic assessment
the assessment programme

31 from building blocks…

32 …to buildings

33 from methods to programmes
► multiple instruments, various formats ► strengths and weaknesses combined ► assessment moments ≠ decision moments

34 every assessment moment is a decision moment
test decision + test decision + test decision = competent

35 every assessment moment is NOT a decision moment
low stakes medium stakes high stakes

36 from methods to programmes
► multiple instruments, various formats ► strengths-weaknesses combined ► assessment moment ≠ decision moment ► multiple quality approaches

37 quality: reliability - consistency - saturation - expertise
- organisation

38 reliability is sampling
short essay2 0.68 0.73 0.84 0.82 practice Video test5 0.68 0.81 0.87 0.90 testing time in hours 1 2 4 8 MCQ1 0.62 0.76 0.93 paper cases1 0.36 0.53 0.69 0.82 orals3 0.50 0.69 0.82 0.90 obser- vation assessment4 0.43 0.60 0.76 0.86 1Norcini et al., 1985 2Stalenhoef-Halling et al., 1990 3Swanson, 1987 4Newble & Swanson, 1987 5Ram et al., 1999

39 generalisability: saturation
orange green blue red yellow purple black nothing new nothing new

40 Sept Overview Dec Mar May June
Yes Mentor agrees with Student Examiner Examiner 1 Ex 1 Ex 2 Final Judgment Examiner 2 Full committee (n=20) Portfolio Committee Assessment Procedure No Sept Dec Mar May June Steps in the year Introduction to students Overview Mentors are trained First portfolio submission No Formative review Examiner training (benchmark portfolios) 2nd portfolio submission Summative review Mentor/student Recommendation F/P/D Exam committee decision

41 Sept Overview Dec Mar May June
Yes Mentor agrees with Student Examiner Examiner 1 Ex 1 Ex 2 Final Judgment Examiner 2 Full committee (n=20) Portfolio Committee Assessment Procedure No Sept Dec Mar May June Steps in the year Introduction to students Overview Mentors are trained First portfolio submission No Formative review Examiner training (benchmark portfolios) 2nd portfolio submission Summative review Here the strategy is prolonged engagement. The student and mentor know each other quite well. There is no issue of doubt on the originality of the work. In the process the mentor acquires a very good picture over the student in relation to the development of his competencies and in relation to the ultimate criteria to be reached. This is highly useful when pass/fail decisions need to be taken, but may harm the relationship between mentor and students. We made a compromise. Mentor/student Recommendation F/P/D Exam committee decision

42 Sept Overview Dec Mar May June
Yes Mentor agrees with Student Examiner Examiner 1 Ex 1 Ex 2 Final Judgment Examiner 2 Full committee (n=20) Portfolio Committee Assessment Procedure No Sept Dec Mar May June Steps in the year Introduction to students Overview Mentors are trained First portfolio submission No Formative review Examiner training (benchmark portfolios) 2nd portfolio submission Summative review The compromise is that the mentor only makes a recommendation, based on a student’s self assessment. The actual decision will be taken by the committee, not the mentor. By asking students to self-assess, to sign for the recommendation or to annotate it, we’ve build in a check about the appropriate judgment of the mentor which also provides useful feedback to the later assessment process. The official strategy is member checking. Mentor/student Recommendation F/P/D Exam committee decision

43 Sept Overview Dec Mar May June
Yes Mentor agrees with Student Examiner Examiner 1 Ex 1 Ex 2 Final Judgment Examiner 2 Full committee (n=20) Portfolio Committee Assessment Procedure No Sept Dec Mar May June Steps in the year Introduction to students Overview Mentors are trained First portfolio submission No Formative review Examiner training (benchmark portfolios) 2nd portfolio submission Summative review There are two feedback cycles in the construction process (actually three, but I won’t go into details). [Those students who fail at the summative review are given another two weeks to remediate.] Anyway, the outcome of the summative review and the recommendation of the mentor should in all likelihood not be a surprise to the student. If it is, something must have gone wrong in the process. Mentor/student Recommendation F/P/D Exam committee decision

44 Sept Overview Dec Mar May June
Yes Mentor agrees with Student Examiner Examiner 1 Ex 1 Ex 2 Final Judgment Examiner 2 Full committee (n=20) Portfolio Committee Assessment Procedure No Sept Dec Mar May June Steps in the year Introduction to students Overview Mentors are trained First portfolio submission No Formative review Examiner training (benchmark portfolios) 2nd portfolio submission Summative review Examiners being mentors themselves are quite informed about the portfolio procedure. They will never judge a portfolio from one of their students. They are trained before the assessment procedure starts and they benchmark their judgments in the portfolio assessment committee against each other. They will be involved in training other mentors next year. The official strategy involved by doing this is called peer examination. Mentor/student Recommendation F/P/D Exam committee decision

45 Sept Overview Dec Mar May June
Yes Mentor agrees with Student Examiner Examiner 1 Ex 1 Ex 2 Final Judgment Examiner 2 Full committee (n=20) Portfolio Committee Assessment Procedure No Sept Dec Mar May June Steps in the year Introduction to students Overview Mentors are trained First portfolio submission No Formative review Examiner training (benchmark portfolios) 2nd portfolio submission Summative review The assessment process itself is a carefully built up sequential procedure of involving more examiners when there is reason to do so. It combines efficiency with thoroughness. On only a few cases a full committee of experts will eventually decide and pass a very thoroughly discussed and motivated final judgment. [click] The official decision is taken by the examination committee of the faculty. Another safeguard here is that the student has the right to appeal. The procedures involved are called stepwise replication and triangulation. Mentor/student Recommendation F/P/D Exam committee decision

46 from methods to programmes
► multiple instruments, various formats ► strengths-weaknesses combined ► assessment moment ≠ decision moment ► multiple quality approaches ► many instruments : many competency domains

47 1 role  1 instrument instruments A B C D domains med expert
scientist domains worker in HCS person

48 multi-modal assessment
instruments med expert scientist domains worker in HCS person

49 from methods to programmes
► multiple instruments, various formats ► strengths-weaknesses combined ► assessment moment ≠ decision moment ► multiple quality approaches ► many instruments : many competency domains ► integrative  holistic not reductionist

50 overview a bit of history programmatic assessment
the assessment programme

51 assignment Build an assessment programme for a workplace-based learning curriculum. GP training practice, assignments plus day-release education 1 year, two semesters, 4 terms supervisor, medical educator, on-line platform

52 design goals and stated purpose programme in action
supporting the programme documenting of the programme improvement approaches to the programme accounting for the programme Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93.

53 If ‘incompetence’ were an illness, how would we diagnose and treat it?

54 design multiple instruments meaningful collation learning focused
self-regulation assessment moment ≠ decision moment longitudinal feasible and efficient

55 purpose

56 safe independent practitioner
medical expert worker in het healthcare system person scholar

57 what is safe?

58 what is safe? mastery + skill + competence +…. self regulation

59 self regulation self driven analyses external information seeking
goal orientation prioritisation realisation/attainment time management 1 Bandura A. social cognitive theory: an agentic perspective. Annual Review Psychology 2001;52:1-26. 2 Dochy F, M.Segers, Sluijsmans D. The Use of Self-, Peer and Co-assessment in Higher Education; a review. Studies in Higher Education 1999;24(3): 3 Eva KW, Cunnington JPW, Reiter HI, Keane D, G N. How can I know what I don't know? Poor self assessment in a well-defined domain. Advances in Health Sciences Education 2004;9:

60 The opposite of good is... …well intended

61 time competing demands costs motivation culture laws perfect assessment program expectations context expertise beliefs understanding patient care

62 relevant research findings
meaningfulness meaningfulness vlieger, het geheugen werkt niet als een harde schijf transfer and domain specificity voorbeeld televisiereparateur deliberate practice oefenen – feedback – oefenen self-regulated learning biases and decision making 1. Posner MI. What is it to be an expert? In: Chi MTH, Glaser R, Farr MJ, editors. The nature of expertise. Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc, 1988:xxix - xxxvi. 2. Schmidt HG, Boshuizen HP. On acquiring expertise in medicine. Special Issue: European educational psychology. Educational Psychology Review 1993;5(3):

63 learning in context a newspaper is better than a glossy magazine
the seashore is better than the street first it is better to run than to walk you will have to try several several times some skills are required but it is easy to learn even small children can enjoy it once successful the risk of complications is minimal birds seldom get too close rain soaks in very fast a rock can serve as an anchor once it breaks loose there is not second chance

64 learning in context: flying a kite
a newspaper is better than a glossy magazine the seashore is better than the street first it is better to run than to walk you will have to try several several times some skills are required but it is easy to learn even small children can enjoy it once successful the risk of complications is minimal birds seldom get too close rain soaks in very fast a rock can serve as an anchor once it breaks loose there is not second chance

65 relevant research findings
meaningfulness transfer and domain specificity meaningfulness vlieger, het geheugen werkt niet als een harde schijf transfer and domain specificity voorbeeld televisiereparateur deliberate practice oefenen – feedback – oefenen self-regulated learning biases and decision making 1. Eva K. On the generality of specificity. Medical Education 2003;37:587-8. 2. Eva KW, Neville AJ, G.R. N. Exploring the etiology of content specificity: Factors influencing analogic transfer and problem solving. Academic Medicine 1998;73(10):s1-5.

66 analogous transfer

67 relevant research findings
meaningfulness transfer and domain specificity deliberate practice meaningfulness vlieger, het geheugen werkt niet als een harde schijf transfer and domain specificity voorbeeld televisiereparateur deliberate practice oefenen – feedback – oefenen self-regulated learning biases and decision making Ericsson KA. An expert-performance perspective of research on medical expertise: the study of clinical performance. Medical Education 2007;41:

68 deliberate practice deliberate practice assessment feedback adjustment

69 feedback concrete constructive focused on improvement ‘connected’
leading to learning goals/learning plans Shute V. Focus on formative feedback. Review of educational research 2008;78(n):

70 loop learning goals activities practice analysis feedback

71 relevant research findings
meaningfulness transfer and domain specificity deliberate practice self-regulated learning meaningfulness vlieger, het geheugen werkt niet als een harde schijf transfer and domain specificity voorbeeld televisiereparateur deliberate practice oefenen – feedback – oefenen self-regulated learning biases and decision making

72 self-regulated learning
phases areas forethought, planning & activation cognition motivation behaviour context cognition motivation monitoring behaviour context cognition motivation control behaviour context cognitions: goals, prior content knowledge and meta cognitive knowledge. Goals: setting and modifying task-specific goals to be used as criteria against which to gauge progress Activating prior knowledge in a planful way (e.g. asking self-questions and prompting) Activation of metacognitive knowledge (learning strategies), procedural knowledge (implementation strategies) and conditional knowledge (when and why to use) Motivations: goal orientations, self-efficacy, perception of difficulty, task value and interest goal orientations: reasons to engage in the task self-efficacy: belief about their capabilities perception of difficulty: how difficult/easy is the task task value: relevance, importance, usefulness interest: personal interest Behaviours: time and effort planning, and planning for self-observation time and effort planning: creating study schedules and allotting time for the various activities planning for self-observation: deciding which methods to use for monitoring progress and regulate behaviours Context: perceptions of the context and the extent to which it can be managed, altered or navigated cognition motivation reaction & reflection behaviour context cf. Schunk DH (2005). Self-regulated learning: The educational legacy of Paul R. Pintrich. Educational Psychologist, 40, 85-94

73 relevant research findings
meaningfulness transfer and domain specificity deliberate practice self-regulated learning reasoning and decision making meaningfulness vlieger, het geheugen werkt niet als een harde schijf transfer and domain specificity voorbeeld televisiereparateur deliberate practice oefenen – feedback – oefenen self-regulated learning biases and decision making 1. Boreham NC. The dangerous practice of thinking. Medical Education 1994;28: 2. Klein G. Naturalistic Decision Making. Human Factors 2008;50(3): 3. Plous S. The psychology of judgment and decision making. New Jersey: McGraw-Hill inc., 1993. 4. Schmidt HG, Machiels-Bongaerts M, Hermans H, ten Cate TJ, Venekamp R, Boshuizen HPA. The Development of Diagnostic Competence: Comparison of a Problem-based, and Integrated, and a Conventional Medical Curriculum. Academic Medicine 1996;71(6):

74 relevant research findings
reliability validity quality frameworks organisational reliability meaningfulness vlieger, het geheugen werkt niet als een harde schijf transfer and domain specificity voorbeeld televisiereparateur deliberate practice oefenen – feedback – oefenen self-regulated learning biases and decision making 1. Williams M, Klamen D, McGaghie W. Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine 2003;15(4): 2. Kane MT. Validation. In: Brennan RL, editor. Educational Measurement. Westport: ACE/Praeger, 2006: 3. Govaerts MJB. Climbing the pyramid; Towards understanding performance assessment. Maastricht University, 2011. 4. Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93

75 term 1 portfolio mid-term end-term direct obs. CCA direct obs.
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 direct obs. CCA direct obs. direct obs. direct obs. direct obs. mcq test MSF mini-releases mini-releases mini-releases mini-releases mini-releases portfolio

76 term 2 portfolio audit audit mid-term end-term direct obs. CCA
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 direct obs. CCA direct obs. direct obs. mcq test mcq test mini-releases mini-releases mini-releases mini-releases mini-releases audit audit MSF portfolio

77 critical case analysis
5 write-ups of real patient consultations relevance analysis learning activities produce exam questions (EMI, KFP, MCQ) increasingly original literature any discussion minuted by registrar

78 directly observed consultations
9 real patient consultations relevance analysis learning goals (practical + theoretical) learning activity demonstration of success in next observed consultation discussion minuted by registrar

79 clinical audit analysis of the practice environment
determination of specific question collection of data draw conclusions describe plan for change + 3 months: look back and annotate any discussion minuted by registrar

80 multiple-choice tests
3 tests of 60 items each blueprinted sit and submit your answers review items, answer key comment on an criticise questions for correctness present in min-release ‘lodge’ appeal against questions score calculation and feedback to registrars

81 mini-releases flexible agenda building informal networks
discuss MCQ test items compile appeal against questions list of ‘informal’ network

82 multi-source feedback
2 times per year nurses, practice manager, receptionist, other practice staff and registrar discussed with supervisor (end-term assessment) and with ME (minuted by registrar) simple form: dealing with tasks, other and yourself simple ordinal scale ample room for qualitative comments

83 mid and end-term assessment
integrative reviewing all the information learning goals and/or remediation plans advisory to performance review committee minuted by registrar

84 portfolio complete dossier including minutes
individual extra information (only if relevant) audit trail basis for future CV or position applications

85 example of a ‘line’ CCA meaning learning test enhanced learning
feedback MCQs analysis test informal/social networks appeal group group appeal transformation research: narratives for feedback feedback

86 design goals and stated purpose programme in action
supporting the programme documentation of the programme improvement approaches to of the programme accounting for the programme Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93.

87 rules and regulation self responsibility comes with accountability (minutes, plagiarism, fraud) focus on learning and remediation information provision to the registrar documentation transparency second opinion/appeals/switch of ME or supervisor organisation reliability/credibility

88 staff development efficiency: short analyses concrete learning goals
focus on learning training of staff (expertise  efficiency) admin support by admin staff division of advisory and decision roles

89 further requirements goals and stated purpose programme in action
supporting the programme: regulations documentation of the programme: ELO, fact sheets, improvement approaches to of the programme: systematic evaluation accounting for the programme: research

90 Thank you


Download ppt "programmatic assessment for learning"

Similar presentations


Ads by Google