1. Intelligent Applications of Learning Theory and Technology to Educational Assessment Chris Dede Harvard University Based on slides by Jim Pellegrino University of Illinois at Chicago

2. Overview of Todays Presentation Understanding educational assessment contributions of cognitive theory and research to assessment design & practice Assessment design & implementation challenges Assessment design & implementation challenges what technology offers in the way of advances and solutions Highlights from the AEL Intelligent Assessment Symposium at the National School Boards Association Conference

3. Highlights from the AEL NSBA Intelligent Assessment Symposium We are in a "reform" movement where powerful methods of teaching/learning are harder to use, due to flawed standards and tests. The only way to improve this situation is to give people something to move towardsnot something to move against, because then well just react away from what we have now into some other flawed method of reform… - Dr. Chris Dede

4. Highlights from the AEL NSBA Intelligent Assessment Symposium …what we know in terms of cognitive science says that assessment has to move beyond what weve been doing for a long time. That is, assessing discreet bits and pieces of knowledge. We have to get at these more complex aspects of knowing and understanding including metacognitive understanding, conceptual organization, and the ways in which students have represented knowledge that helps them solve problems. - Dr. Jim Pellegrino

5. Highlights from the AEL NSBA Intelligent Assessment Symposium There are certain attributes of cognition that transfer across subject matters and then there are certain things that have to be embedded in that subject matter domain…obviously, the more domain independent…the more reusable… Our interest has been trying to find the crosswalkor alignment between and among standards and assessments, standards and instruction, and instruction and assessment. And the reality is, in my view, that nobody knows how to do it very well. - Dr. Eva Baker

6. Highlights from the AEL NSBA Intelligent Assessment Symposium The one thing that No Child Left Behind did, for better or worse, is really help elevate assessments in the debate and discussions about education and education reform. - John Bailey First of all, the truth of the matter is, assessment is hot. The publics attention is on assessment and accountability. It is an incredible opportunity for us to improve what we do. - Dr. Linda Roberts http://www.ael.org/page.htm?&pd=1&index=346

8. Assessment Model of Student Learning in an Academic Domain Observations Interpretation Model Assessment as a Process of Reasoning from Evidence

9. The Assessment Triangle cognition –model of how students represent knowledge & develop competence in the domain observations –tasks or situations that allow one to observe students performance interpretation –method for making sense of the data observation interpretation cognition Must be coordinated!

10. Advances in Sciences of Thinking & Learning: The most critical implications for assessment are derived from study of How People Learn; research on the nature of competence and the development of expertise in specific curriculum domains.How People Learn –Knowledge and understandingKnowledge and understanding –Characteristics of expertiseCharacteristics of expertise –MetacognitionMetacognition –Multiple paths to competenceMultiple paths to competence –Preconceptions and mental modelsPreconceptions and mental models –Situated knowledge and expertise

11. Some Summary Points Contemporary knowledge from the cognitive sciences strongly implies that assessment practices need to move beyond discrete bits and pieces of knowledge to encompass the more complex aspects of student achievement. Instructional programs and assessment practices based on cognitive theory exist for areas of the curriculum. Example in scienceExample in science Cognitive research and theory should be the starting place for assessment design

12. Assessment Design Principles Assessment design should always be based upon a model of student learning and a clear sense of the inferences about student competence that are desired for the particular context of use. Starting with the Student Model: The student model suggests the most important aspects of student achievement that one would want to make inferences about and provides clues about the types of tasks that will elicit evidence to support those inferences.

13. Some Selected Examples of the Use of Student Models Development of number sense (Case & Griffin) Subtraction bugs (Brown & Burton) Intelligent tutors for algebra and geometry (Anderson, Koedinger, Corbett et al.) Facets and misconceptions in physics (Minstrell & Hunt; McDermott) Middle School Mathematics through Applications (Greeno et al.) Australias developmental progress maps (ACER; Masters)

14. If Technology is the Solution, What is the Problem? Problem: Capitalizing on what we know about cognition & measurement to design more powerful and useful assessments –How can we support the types of inferences about student knowledge and competence that are implied by high standards & cognitive research; inferences about knowing and understanding that can help improve the conditions of learning and the outcomes of education. Solution: Make effective use of the many capabilities of technology to meet the major challenges of design and implementation

15. How Technology Can Help Tapping a broader repertoire of cognitive skills and knowledge Going beyond conventional practices for item presentation Implementing a range of task designs and item formats Recording and scoring complex aspects of behavior Embedding assessments in learning environments

16. Assessment Cognition Observations Interpretation Enhancing Linkages in the Assessment Triangle

17. Examples of Enhanced Linkages From Cognition to Observation: Tasks tapping forms of complex knowledge & reasoning concept organization - Concept Mapping Softwareconcept organization complex problem solving scenarios - Architecture and Dental Exams, AmericaQuest environment From Observation to Interpretation: Scoring and interpreting complex aspects of student knowledge and performance Text analysis - LSALSA Complex problem solving - IMMEX IMMEX

18. Connecting Assessment to the Processes of Learning & Instruction Implementing effective formative assessment –intelligent tutors and other software programs like DIAGNOSER, IMMEX, and Summary Street can help teachers assess students learning and interpret the ways in which students solve problems. Assessment embedded in technology- enhanced learning environments –Situations where learning occurs through a process of extended inquiry, often involving collaborative, problem- based activity; environments focused on aspects of the curriculum often difficult to teach –Integration of cognitive theory with instructional design, including embedded assessment practices examples: SMART, Biologica, WISESMARTBiologicaWISE

19. Technology & Assessment Futures Rich sources of information about student learning can be continuously available across wide segments of the curriculum and for individual learners over extended time periods. –Consider a radical shift in how we might do the business of assessment Issue is how to design for this possibility and explore the options it provides for effectively using assessment information to meet the multiple purposes of current assessments and, most importantly, to aid in student learning.

21. Summary of What We Know About the Nature of Expertise Experts have well-organized knowledge: –their knowledge is organized to support understanding (qualitative before quantitative) and it is conditionalized for use. –Experts have fluent access to their knowledge and recognize patterns & chunks. –Such knowledge & expertise is acquired over time and depends on multiple, contextualized experiences. Implications -- Wisdom cant be taught directly and instruction must be directed towards the gradual acquisition of understanding & expertise. Assessments must be designed to capture the multiple and discriminating features of expertise and its emergence.

22. Summary of What We Know About Students Pre-existing Knowledge Students come to the classroom with pre- conceptions about how the world works. If their initial understanding is not engaged they may fail to grasp the new concepts and information that are taught, or they may learn them for purposes of a test but revert to their preconceptions outside the classroom. Implications - Teachers must draw out and work with the preexisting understandings that their students bring with them. Assessment practices must be designed to assist in this process of tapping student mental models.

23. Summary of What We Know About Knowledge & Understanding To develop competence in an area of inquiry, students must: (a) have a deep foundation of factual knowledge, (b) understand facts and ideas in the context of a conceptual framework, and (c) organize knowledge in ways that facilitate retrieval and application. Implication - Teachers must teach some subject matter in depth, providing many examples in which the same concept is at work and providing a firm foundation of factual knowledge. Assessment must be designed to capture the multiple components of organized knowledge.

24. Summary of What We Know About the Importance of Metacognition Expert learners monitor their own state of understanding as they solve a problem, read a passage, or perform a task. Students can be helped to learn to take control of their own learning by defining learning goals and monitoring their progress in achieving them. These Metacognitive skills can be taught but they must be connected to specific content. Implication - The teaching of metacognitive skills should be integrated into the curriculum in a variety of subject areas. Assessment of such skills and strategies is an important concern.

25. Multiple Paths to Competence Not all children learn in the same way or follow the same paths to competence. –childrens problem solving strategies become more effective over time and with practice –the growth process is not a simple, uniform progression, nor is there movement directly from erroneous to optimal solution strategies. Implication - Assessments should identify specific strategies children are using and where they fall on a continuum of efficiency and appropriateness for a given domain.

26. Mental Models & Misconceptions: An Example from H. S. Science Jim Minstrells work on mapping FACETS of knowledge and understanding in multiple areas of Physics –DIAGNOSER program for use in instruction. A facet is a convenient unit of thought, an understanding or reasoning, a piece of content knowledge or strategy seemingly used by the student in making sense of a particular situation. (Minstrell, 1992, pg. 2)

28. Scale reading = 10.0 lbsScale reading = _____ lbs Example Diagnostic Item Glass case with air removed

29. A Facet Cluster for Separating Medium from Gravitational Effects 310 - pushes from above and below by a surrounding fluid medium lend a slight support (Pretest = 3%) 311 - a mathematical formulaic approach (e.g., rho x g x h1 - rho x g x h2 = net buoyant pressure) 314 - surrounding fluids dont exert any forces or pushes on objects 315 - surrounding fluids exert equal pushes all around an object (Pre = 35%) 316 - whichever surface has greater amount of fluid above or below the object has the greater push by fluid on the surface 317 - fluid mediums exert an upward push only (Pretest = 13%) 318 - surrounding fluid mediums exert a net downward push (Pre = 29%) 319 - weight of an object is directly proportional to medium pressure on it (Pretest = 20%)

35. Source: White & Gunstone: Probing Understanding, 1992, p. 16. water rain rivers clouds soil sun oceans is falling water comes from contain goes into rivers shines on flow to contain

38. Adventures of Jasper Woodbury