Criteria for Good Assessment John Norcini, Ph.D.

Overview Some background What is a test or assessment? How does assessment affect learners? How are assessments changing? Who cares about assessment? Characteristics of good assessment How do they relate to purpose? How do they relate to the stakeholders?

What is a test or assessment? Two definitions of test To discover the worth of something by trial Summative, evaluation To improve the quality of something by trial Formative, assessment For today, a test or assessment is any measurement process Formative or summative These are central to Understanding the importance of assessment Understanding the application of assessment Identifying criteria for good assessment

How does assessment affect learners? Assessment is a powerful tool in education Motivates students to learn Directs learning by signaling what it important Unintended effects when there is a mismatch between curriculum and assessment Ensures that learning has occurred Creates learning

How are assessments changing? Past That part of eternity with some small fraction of which we have a slight and regrettable acquaintance. A Bierce Little realism Little person/patient involvement Lack of structure during workplace training Episodic and infrequent continuing education Future That period of time in which our affairs prosper, our friends are true, and our happiness is assured. A Bierce Authenticity Active person/patient involvement More structured workplace training Life-long learning and quality improvement

Who cares about assessment? Variety of stakeholders Students Key to learning and demonstration of competence Teachers and individual/collective educational institutions Accountability and self-assessment Person/patients and healthcare systems Accountability and efficiency Regulatory bodies Accountability and maintenance of competence

Characteristics of good assessment No single set of criteria apply equally to all assessment situations Different purposes may lead to different values for the criteria Different stakeholders and different mixes of stakeholders may value the criteria differently Ottawa conference produced a set of criteria and how they are modified by purpose and stakeholders

Characteristics Validity or coherence Is there a body of evidence that hangs together and supports the use of assessment for a particular purpose Is a property of the inferences drawn from a test, not the test itself Is a matter of degree Requires the ongoing collection of data In seeking absolute truth we aim at the unattainable, and must be content with finding broken portions. William Osler, 1889

Characteristics Reliability or reproducibility Will scores be the same if examinees are reassessed? History of reliability Test-retest reliability Alternate form reliability Split-half reliability Reliability index Arithmetic…is a bad master, but it is a good servant. Anonymous Lancet, 1921

Characteristics Equivalence Different versions of an assessment yield equivalent scores or decisions A challenge for assessment in the workplace Catalytic effect The assessment provides results and feedback in a fashion that enhances learning A requirement for formative assessment

Characteristics Educational effect The assessment motivates those who take it to prepare in a fashion that has educational benefit How do students prepare for the test? Acceptability Stakeholders find the assessment process and results to be credible Feasibility The test is practical, realistic, and sensible, given the circumstances and context

Characteristics and Purpose FormativeSummative Validity-coherence+++++ Reproducibility-consistency++++ Equivalence++++ Educational effect++ Catalytic effect++++ Feasibility++ Acceptability++

Characteristics and Stakeholders Examinees-FormativeExaminees-Summative Validity-coherence+++++ Reproducibility-consistency++++ Equivalence++++ Educational effect++++ Catalytic effect++++ Feasibility++ Acceptability+++++

Characteristics and Stakeholders Teacher-Institution Formative Teacher-Institution Summative Validity-coherence+++++ Reproducibility-consistency++++ Equivalence++++ Educational effect++ Catalytic effect++++ Feasibility++ Acceptability++

Characteristics and Stakeholders Persons/patientsHealthcare System Validity-coherence+++ Reproducibility-consistency+++ Equivalence+++ Educational effect++ Catalytic effect+++ Feasibility+++ Acceptability++

Characteristics and Stakeholders Regulators Validity-coherence+++ Reproducibility-consistency+++ Equivalence+++ Educational effect+ Catalytic effect++ Feasibility++ Acceptability++

Summary Characteristics of good assessment Validity, reliability, equivalence, catalytic effect, educational effect, acceptability, feasibility Influenced by the purpose Formative and summative Influenced by the stakeholders Learners, teachers-institutions, patients-healthcare system, regulators