The Role of Information in Systems for Learning Paul Nichols Charles DePascale The Center for Assessment

Problem Recognition that improved student learning requires coordinated assessment, instruction, curricula and professional development States and districts are being asked to conceptualize and implement assessments as a component of broad and coherent systems For example, comprehensive assessment systems or systems for learning. New burdens on district and state staff

Problem Think beyond a single test or set of tests to coordinated system of assessment, curriculum, pedagogy and professional development Manage and held accountable for the award, development and implementation of multi- component systems Cross department boundaries, engage multiple- vendors and involve many times the cost of a testing program

Status Quo State and district staff already have or can find the following training and tools to help with test development and implementation for individual tests Currently no theoretical framework and no indices to help state, district and other educational leaders in planning, developing and evaluating systems for learning

Systems Approach Move beyond hand waving to offer a framework and quantitative tools used by district and state staff to conceptualize and implement systems for learning (SFL) Note the components that probably should be included in an SFL Address SFL coherence by looking at communication of information from students to teachers Present analytic framework allowing district and state staff to predict SFL coherence across different configurations

Defining Systems for Learning What is a system? From the field of systems thinking: A system is “a collection of parts which interact with each other to function as a whole” An SFL is a set of components which interact with each other to function as a whole with the intention of improving student learning

Defining Systems for Learning Paul Black and colleagues describe the components of an SFL:

Defining Systems for Learning Curriculum consists of the knowledge and skills in subject areas that teachers teach and students learn. Instruction refers to methods of teaching and the learning activities used to help students master the content and objectives specified by a curriculum. Assessment is the means used to measure the outcomes of education and the achievement of students with regard to important competencies. Theory of learning is the model of student cognition and learning in the domain

Defining Systems for Learning Karin Hess reminds us that professional development is an important missing component:

Defining Systems for Learning Coherence is provided by the theory of learning The components must “talk to each other” in a common language The language of communication is the learning theory

Measuring SFL Coherence An SFL may be conceptualized as a communication system Think of a radio or your iPad receiving a transmission A generic communication system

Measuring SFL Coherence Signal is generated by an information source Transmitter codes the signal and passes the coded signal to the receiver Receiver decodes the signal and passes the reconstructed signal to the destination Noise is likely to degrade the information in the signal as it is passed from component to component Information is contaminated by noise so that information received is different from an error- free message

Measuring SFL Coherence Teacher attempting to understand the information relevant to the construct in student performance

Measuring SFL Coherence From Measurement Theory to Information Theory Information is defined as the amount of uncertainty reduced – Student whose answer is right or wrong vs student whose answer reveals stage of learning progression Bit is measure of amount of information in terms of choice or uncertainty Learning outcome between two equally probable alternatives is learning one bit of information

Measuring SFL Coherence Coherence can be defined as the degree to which different components help communication H(x|y), equivocation, is uncertainty that x was transmitted when y is received H(x|y) = Mapping from x to y reduces noise or error the value approaches 0 Lower values = higher coherence

Measuring SFL Coherence Components hinder/facilitate communication so decrease/increase coherence – Performance-based assessment where the teacher is looking for formative information – Scaffolding reading or constructed response on the circulatory system – Learning progression with 5 levels – Focus on professional development & instruction

Measuring SFL Coherence Simulation 1: Professional development addresses basic statistics, reliability, validity Fails to address learning progressions Teachers’ scoring shows no relationship to students’ learning progression stage Students receive instruction unrelated to prior understanding Coherence low, H(x|y) = 2.32

Measuring SFL Coherence Simulation 2: Professional development addresses the concept of learning progressions and describes stages in the learning progression Teachers’ scoring shows large “halo effect” and noise Students with impoverished understanding fail to receive instruction Coherence moderate, H(x|y) = 1.49

Measuring SFL Coherence Simulation 3: Professional development addresses concept of learning progressions, describes stages and in the learning progression, calibrates scoring with exemplars Teachers’ scoring shows some “halo effect” and impact of noise Most students receive appropriate instruction H(x|y) = 0.83

Conclusion State and district staff are asked to conceptualize and implement SFLs State and district staff need a theoretical framework and quantitative indices to help plan and evaluate SFLs Measurement theory focuses on individual tests or groups of tests Instructional theory focuses on instructional techniques and activities The framework and index of system coherence are a start…