1. Vurdering og læring – får vi det til? Assessment and Learning – Fields Apart? Summary and Comments Jan-Eric Gustafsson Professor II

2. Purposes of assessment of – for – as learning at individual – institutional – system levels Assessments typically serve multiple purposes and purposes often change over time Different stakeholders ascribe different purposes to the same assessment The validity of an assessment must be evaluated against its purpose(s).

3. Validity Amazing development of neo- and post-Messick models and of argumentation-based models The chain is only as strong as its weakest link The generalisation link: Strong in standardized, itemized, assessments; weak in complex, authentic assessments The extrapolation link: Strong in complex, authentic assessment; weak in standardized, itemized, assessments If generalisation and extrapolation are both important, validity is difficult to achieve

4. Methodological approaches Much methodological debate starts from a dichotomy between quantitative and qualitative methods. However, Ercikan and Roth (2006) argued that this dichotomy is fallacious: – Quantitative research is typically based on qualitative distinctions in data generation and in conceptualisation – Much qualitative research aims at, and does achieve, generalizations They proposed instead a continuous scale that goes from the lived experience of people on one end (low-level inference) to idealized and generalized patterns of human experience on the other (high-level inference). – Low-level inference research is characterized by contingency, particularity, being affected by the context, and concretization, – High-level inference research is characterized by standardization, universality, distance, and abstraction.

5. A metaphor for low- and high-level inference research: weather and climate Weather affects our daily lives, how we dress, what we do and talk about. We may adapt to weather but there is not much we can do about it. In the short run we can predict weather, but beyond a week or so weather is unpredicatble. Climate is generalized weather over a longer period of time. We experience weather, and through aggregating these experiences, we get a sense of climate. In a more precise manner scientists define climate as aggregate aspects of weather, using indicators such as mean temperature and mean rainfall. Thus, climate is an abstraction. While weather is unpredictable and chaotic, climate and climate changes are stable phenomena, which we can be understand theoretically and for which empirically based models may be constructed, that predict long-term development. In terms of this metaphor, high-level inference research is concerned with climate, while low-level inference research is concerned with weather.

6. The foundation of high-level inference research: aggregation Climate is a social construction, and research on climate is based on a highly developed technology of devices for generating data, on agreed-upon definitions, and analytical models. But the fundamental idea is to aggregate multiple observations of different aspects of weather. In the same manner quantitative research in education is based on aggregation of observations of different aspects of phenomena of teaching and learning: Two types of aggregation: – aggregation over observational units, such as students, classes, schools, municipalities and school-systems. – aggregation over different observations for the same unit, such as when students are followed and observed over longer periods of time.

7. High-level inference procedures Aggregation of observations is used to create measures of abstract constructs, such as reading literacy, end-of-term grades, or a licensing assessment. With aggregation over observations the generalization link is strengthened, according to the general principle that combination of fallible observations from different contexts and contents yields a more generalizable score However, agggregation over contexts and contents weakens the extrapolation link because the more abstract measure is not easily connected to such real- life situations that we want to make extrapolations to.

8. When can we take advantage of high-level inference procedures? In construction of standardized tests of and for learning In class-room based assessment of learning over longer periods of time In aggregation of student responses to institutional and system levels

9. When can we take advantage of low-level inference procedures? In class-room based assessment for learning, focussing on specific content and tasks (teacher assessment, supported by qualitative information and high pedagogical content knowledge)

10. Some conclusions When generalization is of little importance, systematic low-level inference procedures support assessment for learning and short-term assessment of learning. When generalisation is important high-level inference procedures are essential

11. How does assessment support learning? Through assessment for learning at individual, institutional and system levels Through assessment as learning, developing student metacognition and knowledge how to monitor their own learning Through intended and unintended impact on teaching and learning However, we need to be aware of the problem of multiple purposes, and the risk of corruption caused by high stakes

12. Thank you very much for your attention!