Moving beyond the psychometric discourse: A model for programmatic assessment Researching Medical Education Association for the Study of Medical education London, 23 November 2010 Cees van der Vleuten Maastricht University The Netherlands Powerpoint:

The first step is to measure whatever can be easily measured. This is ok as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide. —Charles Handy, The Empty Raincoat, page 219.

 The Mc Namara fallacy refers to Robert McNamara, The Secretary of defence from and his belief that the body count was a good way of measuring how the war was going.  As long as more Vietcong were being killed than US forces, the war was being won

 Planned arrangement of individual assessments in a learning program  Quality compromises are made for individual methods/data points but not for the program as a whole  Programmatic assessment is fit for purpose: ◦ Assessment for learning ◦ Robust decision making over learner’s performance ◦ Improving the learned curriculum

 Proposed in 2005 (Van der Vleuten & Schuwirth, 2005)  Multitude of quality criteria and self- assessment instrument of program quality (Baartman et al, 2006, 2007)  Framework of design (Dijkstra et al, 2009) and design guidelines (Dijkstra et al, in prep)  Need for theoretical model of programmatic assessment for program in action

 Any point measurement or single data point of assessment is flawed  Standardized methods of assessment can have ´built-in´ validity ◦ Quality control around test construction and administration ◦ Assessment ‘technology’ is available  Validity in unstandardized methods lies in the users of the instruments, more than in the instruments 1 Theoretical/empirical account: Van der Vleuten, C. P., Schuwirth, L. W., Scheele, F., Driessen, E. W., & Hodges, B. (2010). The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol, 24,

 Assessment drives learning ◦ Requires richness or meaningfulness of data ◦ Qualitative, narrative information carries a lot of weight ◦ Theoretical understanding emerging (Cilliers et al., 2010, under editorial review)  S takes (formative and summative assessment) is a continuum  N of datapoints need to be proportional to stakes  Expert judgment is imperative for assessing complex competencies and when diverse information is to be combined ◦ Sampling strategies can reduce random error ◦ Procedural strategies can reduce bias 1 Van der Vleuten et al., 2010

Time Assessment Activities Training Activities Supporting Activities Artifacts of learning - Outcome artifacts: Products of learning tasks - Process artifacts: Learning or working activities vv Learning task - PBL case - Patient encounter - Operation - Project - Lecture - Self-study

Time Assessment Activities Training Activities Supporting Activities Individual data points of assessment - Fit for purpose - Multiple/all levels of Miller - Learning oriented, Information rich documentation, meaningful (quantitative, qualitative) - Low stake Certification of mastery-oriented learning tasks - Rescuscitation - Normal delivery of infant vv

Time Assessment Activities Training Activities Supporting Activities Supportive social interaction - Coaching/mentoring/supervision - Peer interaction (intervision) (P)Reflective activity by learner - Interpretation of feedback - Planning new learning objectives and tasks vv

Time Assessment Activities Training Activities Supporting Activities Intermediate evaluation - Aggregate information held against performance standard - Committee of examiners - Decision making: diagnostic, therapeutic, prognostic - Remediation oriented, not repetition oriented - Informative - Longitudinal - Intermediate stake vv Firewall dilemma - Dilemma between access to rich information and compromising relationship supporting person(s) and learner

Time Assessment Activities Training Activities Supporting Activities vvvv

Time Assessment Activities Training Activities Supporting Activities vvvvvv Final evaluation - Aggregate information held against performance standard - Committee of examiners - Pass/fail(/distinction) high stake decision - Based on many data points and rich information - Decision trustworthiness optimized though procedural measures, inspired qualitative methodology strategies - High stake

QuantitativeQualitative Criterionapproachapproach Truth valueInternal validityCredibility ApplicabilityExternal validityTransferability ConsistencyReliabilityDependability NeutralityObjectivityConfirmability

Strategy to establish trustworthinessCriteria Potential Assessment Strategy (sample) CredibilityProlonged engagementTraining of examiners TriangulationTailored volume of expert judgment based on certainty of information Peer examinationBenchmarking examiners Member checkingIncorporate learner view Structural coherenceScrutiny of committee inconsistencies TransferabilityTime samplingJudgment based on broad sample of data points Thick descriptionJustify decisions DependabilityStepwise replicationUse multiple assessors who have credibility ConfirmabilityAuditGive learners the possibility to appeal to the assessment decision

vv Time Assessment Activities Training Activities Supporting Activities vvvv

vv Time Assessment Activities Training Activities Supporting Activities vvvv vv Time Assessment Activities Training Activities Supporting Activities vvvv

 Resources/cost (do fewer things well rather than doing more but poorly)  Bureaucracy, Trivialization, Reductionism  Legal restrictions  Novelty/unknown

 Assessment for learning combined with rigorous decision making  Post-psychometric era of individual instruments  Theory driven assessment design  Infinite research opportunities ◦ Multiple formalized models of assessment (e.g. psychometrics, Bayesian approaches to information gathering, new conceptions of validity……) ◦ Judgment (bias, expertise, learning….) ◦ How and why is learning facilitated (theory of assessment driving learning) ◦ ………..