1. Knowing What Students Know Ganesh Padmanabhan 2/19/2004

2. Overview Nature of Assessment Nature of Assessment Need for Reform Need for Reform Advances in Cognition Advances in Cognition Advances in Measurement Advances in Measurement Role of Technology Role of Technology

3. Assessment “Assessment is the means used to measure the outcomes of education and achievement of students with regard to important competencies.” “Assessment is the means used to measure the outcomes of education and achievement of students with regard to important competencies.” Purpose determines priorities. Purpose determines priorities. Context of use imposes constraints on design. Context of use imposes constraints on design.

4. Purposes of Assessment Achievement vs Aptitude Achievement vs Aptitude Formative vs Summative Formative vs Summative Evaluation of Educational Programs Evaluation of Educational Programs More and more high-stakes decisions More and more high-stakes decisions One type does not fit all One type does not fit all Need for better alignment Need for better alignment

5. Reasoning From Evidence “Assessment is a tool designed to observe students’ behavior and produce data that can be used to draw reasonable inferences about what the students know.” “Assessment is a tool designed to observe students’ behavior and produce data that can be used to draw reasonable inferences about what the students know.” Also inferences about how, when, and whether they use what they know. Also inferences about how, when, and whether they use what they know. Imprecise by-nature Imprecise by-nature

6. Assessment Triangle

7. Cognition “…refers to a theory or set of beliefs about how students represent knowledge and develop competence in a subject domain.” “…refers to a theory or set of beliefs about how students represent knowledge and develop competence in a subject domain.” Cognitive Theories Cognitive Theories Educational Theories Educational Theories Experience of Expert Teachers Experience of Expert Teachers Need not be exhaustively complex Need not be exhaustively complex

8. Observation “Every assessment is also based on a set of beliefs about the kinds of tasks or situations that will prompt students to say, do, or create something that demonstrates important knowledge and skills.” “Every assessment is also based on a set of beliefs about the kinds of tasks or situations that will prompt students to say, do, or create something that demonstrates important knowledge and skills.” Not arbitrary, needs careful design. Not arbitrary, needs careful design.

9. Interpretation “…expresses how the observations derived from a set of assessment tasks constitute evidence about the knowledge and skills being assessed.” “…expresses how the observations derived from a set of assessment tasks constitute evidence about the knowledge and skills being assessed.” Methods for drawing inferences Methods for drawing inferences Classroom – usually qualitative Classroom – usually qualitative Large-Scale – usually formal, statistical Large-Scale – usually formal, statistical

10. Making Sound Inferences Depends on making explicit connections in assessment triangle between Depends on making explicit connections in assessment triangle between –Cognition and Observation –Cognition and Interpretation –Observation and Interpretation –Interdependent and Iterative Development

11. Advances in the Learning Sciences “How People Learn” by NRC (1999) “How People Learn” by NRC (1999) Systematic investigation dates back to the late 19 th Century Systematic investigation dates back to the late 19 th Century Recently, very diverse fields converging Recently, very diverse fields converging “Cognitive Revolution” (1960s onward) “Cognitive Revolution” (1960s onward) Four significant perspectives: Differential, Behaviorist, Cognitive, and Situative Four significant perspectives: Differential, Behaviorist, Cognitive, and Situative

12. Differential Perspective “The differential perspective focuses mainly on the nature of individual differences in what people know and in their potential for learning.” “The differential perspective focuses mainly on the nature of individual differences in what people know and in their potential for learning.” Early 1900s Early 1900s Bell Curve Assumptions Bell Curve Assumptions Focus on aptitude not achievement Focus on aptitude not achievement

13. Behaviorist Perspective Skills are thought to be composed of stimulus-response associations. Skills are thought to be composed of stimulus-response associations. Considers learning as the process by which one acquires those associations and assembles them into skills. Considers learning as the process by which one acquires those associations and assembles them into skills. Stimulus, reinforcement, conditioning theories etc. Stimulus, reinforcement, conditioning theories etc.

14. Cognitive Perspective “In cognitive theory, knowing means more than the accumulation of facts and knowledge; it means being able to integrate knowledge, skills, and procedures in ways that are useful for interpreting situations and solving problems.” “In cognitive theory, knowing means more than the accumulation of facts and knowledge; it means being able to integrate knowledge, skills, and procedures in ways that are useful for interpreting situations and solving problems.” Aids assessment in figuring how, when, and whether students use what they know. Aids assessment in figuring how, when, and whether students use what they know.

15. Situative Perspective Sociocultural View Sociocultural View Places individual thinking within a practical context Places individual thinking within a practical context Authentic Assessment Authentic Assessment Needs more here. Needs more here.

16. Interpreting Evidence Psychometrics Psychometrics Probabilistic Approach to Reasoning Probabilistic Approach to Reasoning Cognition  Construct Cognition  Construct Observation  Observational Model Observation  Observational Model Interpretation  Measurement Model Interpretation  Measurement Model

17. Student Construct Observations Generation of Data Interpretation of Data x ө “…statistical models can be developed to predict the probability that people will behave in certain ways in assessment situations, and that evidence derived from observing these behaviors can be used to draw inferences about students’ knowledge, skills, and strategies…”

18. Classical Test Theory ө Sum of Responses Construct Observations

19. ө Generalizability Theory Sum of Responses Observations Construct RatersType of Task

20. Item Response Modeling ө Construct Response 1Response 2Response 3Response 4Response 5 δ1δ1 δ2δ2 δ3δ3 δ4δ4 δ5δ5 Observations

21. Latent Class Models Fig 4-8 Fig 4-8

22. Multi-Attribute Models Fig 4-9 Fig 4-9

23. Role of Technology Theory-Based Item Generation Theory-Based Item Generation Concept Organization Concept Organization Complex Problem Solving (e.g. IMMEX) Complex Problem Solving (e.g. IMMEX) Text Analysis and Scoring Text Analysis and Scoring Portal computer-aided approach for design of assessment Portal computer-aided approach for design of assessment

24. Questions

25. References All quotes and figures were taken from the NRC Report “Knowing What Students Know” by Pellegrino et al. (2001)