1. Designing an Assessment System for the Race to the Top Edward Haertel School of Education Stanford University RTTT Public Meeting, General Assessment Denver, Colorado December 1, 2009 1

2. What’s Different This Time? In the absence of well-defined and demanding standards, education in the United States has gravitated toward de facto national minimum expectations, with curricula focusing on low-level reading and arithmetic skills and on small amounts of factual material in other content areas. Most current assessment methods reinforce the emphasis on these low- level skills and processing bits of information rather than on problem solving and critical thinking. The adoption of world- class standards would force the Nation to confront today's educational performance expectations that are simply too low. NCEST Report, 1992 2

3. New Ideas Common Standards ◦ fewer, clearer, higher? Aligned assessments of individual growth Explicit attention to range of assessment types / formats, content sampling, item release plans, etc. 21 st century technology ◦ computer-based testing, longitudinal growth models, Evidence-Centered Design, … 3

4. Old Ideas If assessments are aligned to standards, then growth in test scores will track intended schooling outcomes Essentially all valued schooling outcomes can be captured by test performances of ◦ students working alone ◦ without “smart workplace” resources ◦ answering clear questions with clear answers 4

5. Old Ideas One assessment system will “do it all” ◦ track progress  of students / schools / … ◦ inform decisions  about graduation / college readiness / career readiness / college course placement ◦ support improvement  in teaching / learning / instructional programs ◦ determine effectiveness  of schools / principals / teachers 5

6. 6 Theories of Action are, if anything, less clear than in 1992 A Theory of Action is a chain of logical propositions that “connects the dots,” explaining step by step, in commonsense terms, how the assessment system, accountability system, and related reforms will lead to improved learning outcomes

7. Rethinking Old Ideas To Make New Ideas Work Aligning tests and curricula to common standards is not sufficient to create a coherent system 7 Content Standards Curriculum & Instruction Assessments ?

8. Rethinking Old Ideas To Make New Ideas Work Aligning tests and curricula to common standards is not sufficient to create a coherent system 8 Content Standards Curriculum & Instruction Assessments ?

9. 11th Grade History / Social Studies US11.11.2. Discuss the significant domestic policy speeches of Truman, Eisenhower, Kennedy, Johnson, Nixon, Carter, Reagan, Bush, and Clinton (e.g., education, civil rights, economic policy, environmental policy). 9

10. Item Testing US11.11.2 10

11. High School Biology BI6.f Students know at each link in a food web some energy is stored in newly made structures but much energy is dissipated into the environment as heat. This dissipation may be represented in an energy pyramid. 11

12. Item Testing BI6.f 12

13. Rethinking Alignment It is not enough for each item to be somehow connected to one or more standards Assessments, standards, and curriculum and instruction must all be considered together ◦ Curriculum-embedded formative assessments ◦ Portfolios incorporating well-specified, externally scorable components 13

14. Rethinking Old Ideas To Make New Ideas Work Testing need not imply that students work alone, without references or resources, on small tasks with one right answer ◦ Well-structured problems (Frederiksen, 1984) ◦ Content-lean, process-constrained tasks (Baxter & Glaser, 1998) ◦ Sequestered Problem Solving (Bransford & Schwartz, 1999) 14

15. Rethinking What It Means to Take A Test A comprehensive assessment system must also include ◦ ill-structured problems with multiple solutions that are better or worse, not right or wrong ◦ content-rich, process-open tasks requiring flexible, adaptive application of coherent knowledge structures to solve novel problems ◦ tests of preparation for future learning – measuring skill in approaching new problems, “knowing with” as well as “knowing how” or “knowing that” 15

16. Rethinking Old Ideas To Make New Ideas Work A single form of testing cannot satisfy the full set of RTTT design requirements ◦ Sole reliance on external, on-demand tests will fall short, even if constructed-response items are included ◦ Sole reliance on external, on-demand tests will fall short, even if interim / benchmark assessments are included 16

17. Rethinking Multiple Indicators A comprehensive assessment system must use multiple forms of evidence ◦ curriculum-embedded assessments ◦ extended projects ◦ a range of student performances including  finding, organizing, and evaluating information  analyzing and synthesizing data  planning and carrying out investigations  solving unfamiliar, open-ended problems  written reports and oral presentations 17

18. Rethinking Multiple Indicators Different forms of assessment, organized in different ways, may be required for ◦ improving instructional decision making in the classroom ◦ tracking student growth and predicting college or career readiness ◦ accountability 18

19. Rethinking Multiple Indicators Standards, assessments, curriculum, and instruction must be aligned and coherent ◦ horizontal links across content areas ◦ vertical links across grade levels Assessment at the classroom or school level may employ matrix sampling ◦ different students answer different questions to provide better group-level information ◦ some assessment system components may not yield individual scores 19

20. Summary We must RETHINK Alignment ◦ not just test-to-standards, but instead a coherent, simultaneous consideration of standards, assessments, curriculum and instruction, and I might add, teacher preparation 20

21. Summary We must RETHINK what it means to take a test ◦ not just sequestered problem solving, with tidy questions and routine problems, but also complex tasks  with multiple solutions judged by multiple criteria  sometimes worked out over an extended time  sometimes involving collaboration  sometimes requiring students to figure out what information they need and where to find it 21

22. Summary We must RETHINK multiple indicators ◦ not just a broader mix of item formats ◦ not just interim plus end-of-year tests ◦ some portion of classroom assessment must be sufficiently documented and structured so as to be incorporated into comprehensive assessment systems to meet multiple needs 22

23. 23 Thank you