1. 1 Assessments to support deeper learning and ambitious instruction P. David Pearson UC Berkeley Slides will be posted at www.scienceandlitercy.org

2. OR On-demand performance assessment: The good, the bad, the ugly, and the incredibly beautiful! On-demand performance assessment: The good, the bad, the ugly, and the incredibly beautiful! 2

3. 3 Assessing Literacy: Problems and Possibilities??? How did we get to where we are? How did we get to where we are? Where are we headed? Where are we headed? Where should and could we be headed? Where should and could we be headed? P. David Pearson P. David Pearson UC Berkeley UC Berkeley www.scienceandliteracy.org www.scienceandliteracy.org

4. 4 Some Context for Today’s Talk My interest stems from several sources My interest stems from several sources Multiple attempts to build state assessment systems Multiple attempts to build state assessment systems Several years of attempts to build alternative assessments—portfolios, performance assessments, formative systems. Several years of attempts to build alternative assessments—portfolios, performance assessments, formative systems. Attempts to work with schools to build assessment systems rather than collections of tests Attempts to work with schools to build assessment systems rather than collections of tests A conviction that accountability should not drive us into a corner in which we engage in practices that are not in the best interests of students, teachers, or the public. A conviction that accountability should not drive us into a corner in which we engage in practices that are not in the best interests of students, teachers, or the public.

5. 5 Part I How did we get to where we are? Where did it all start? Early 1960s: A kinder, gentler time Early 1960s: A kinder, gentler time Title 1 reauthorization in 1967: the first trade of accountability for flexibility Title 1 reauthorization in 1967: the first trade of accountability for flexibility

6. 6 The 1970s Behavioral objectives Behavioral objectives Criterion referenced assessments Criterion referenced assessments Curriculum-embedded assessments Curriculum-embedded assessments Minimal competency tests: New Jersey Minimal competency tests: New Jersey Statewide assessments: Michigan & Minnesota Statewide assessments: Michigan & Minnesota

7. 7 Skill 1 Skill 2 Teach Assess Conclude Teach Assess Conclude The 1970s Skills management mentality: Teach a skill, assess it for mastery, reteach it if necessary, and then go onto the next skill. Historical relationships between instruction and assessment Foundation: Benjamin Bloom’s ideas of mastery learning

8. 8 Skill 1 Skill 2 Skill 3 Teach Assess Conclude Teach Assess Conclude Teach Assess Conclude Skill 4 Skill 5 Skill 6 Teach Assess Conclude Teach Assess Conclude Teach Assess Conclude The 1970s, cont. And we taught each of these skills until we had covered the entire curriculum for a grade level.

9. 9 Rule of Thumb The finer the grain size at which we monitor a process like reading and writing, the greater the likelihood that we will end up teaching and testing bits and pieces rather than global processes like comprehension and composition. The finer the grain size at which we monitor a process like reading and writing, the greater the likelihood that we will end up teaching and testing bits and pieces rather than global processes like comprehension and composition.

10. 10 The 1980s In like a lamb, out like a lion In like a lamb, out like a lion Quiet consolidation of skills based learning (especially in commercial materials) Quiet consolidation of skills based learning (especially in commercial materials) A Nation at Risk leads to call for school reform A Nation at Risk leads to call for school reform The evolution of “higher order” thinking in all subject matter areas The evolution of “higher order” thinking in all subject matter areas Late 1980s: Do we have assessments to match our models of learning---------- Late 1980s: Do we have assessments to match our models of learning---------- Performance and authentic assessment movements Performance and authentic assessment movements Eradicate the distinction between instruction and assessment Eradicate the distinction between instruction and assessment

11. Key movements of the late 1980s/1990s Kentucky and Vermont portfolio assessments Kentucky and Vermont portfolio assessments Graduation by portfolio presentation Graduation by portfolio presentation Central Park East Central Park East Walden Walden Assessment by exhibition: Coalition Assessment by exhibition: Coalition Performance Assessment: New Standards, CLAS, Maryland… Performance Assessment: New Standards, CLAS, Maryland… Some models of teacher performance assessment: NBPTS, Connecticut Some models of teacher performance assessment: NBPTS, Connecticut 11

12. 12 Late 1980s/early 1990s: Portfolios Performance Assessments Make Assessment Look Like Instruction On standards 1-n Activities Conclusions From which we draw We engage in instructional activities, from which we collect evidence which permits us to draw conclusions about student growth or accomplishment on several dimensions (standards) of interest.

13. 13  The complexity of performance assessment practices: one to many Any given activity may offer evidence for many standards, e.g, responding to a story. Activity X Standard 5 Standard 3 Standard 4 Standard 2 Standard 1

14. 14 Standard X Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 For any given standard, there are many activities from which we could gather relevant evidence about growth and accomplishment, e.g., reads fluently  The complexity of performance assessment practices: many to one

15. 15  The complexity of portfolio assessment practices, many to many Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 Any given artifact/activity can provide evidence for many standards Any given standard can be indexed by many different artifacts/activities

16. 16  Some people can tell what time it is by looking at the sun, but I never have been able to make out the numbers.  There are four seasons: salt, pepper, mustard, and catsup. The perils of performance assessment: or maybe those multiple-choice assessments aren’t so bad after all…….

17. 17 "Water is composed of two gins, Oxygin and Hydrogin. Oxygin is pure gin. Hydrogin is gin and water." "Water is composed of two gins, Oxygin and Hydrogin. Oxygin is pure gin. Hydrogin is gin and water." The perils of performance assessment

18. 18 "Germinate: To become a naturalized German." "Germinate: To become a naturalized German." "Vacumm: A large, empty space where the pope lives." "Vacumm: A large, empty space where the pope lives." The perils of performance assessment

19. 19  Genetics explains why you look like your father, and if you don't, why you should. The perils of performance assessment

20. 20 You can listen to thunder and tell how close you came to getting hit. If you don't hear it, you got hit, so never mind. You can listen to thunder and tell how close you came to getting hit. If you don't hear it, you got hit, so never mind. "When you breath, you inspire. When you do not breath, you expire." "When you breath, you inspire. When you do not breath, you expire." The perils of performance assessment

21. 21 1990s Standards based reform Standards based reform State initiatives State initiatives IASA model IASA model Trading flexibility for accountability Trading flexibility for accountability Move from being accountable for the means and leaving the ends up for grabs (doctor or lawyer model) TO Move from being accountable for the means and leaving the ends up for grabs (doctor or lawyer model) TO Being accountable for the ends and leaving the means up for grabs (carpenter or product model) Being accountable for the ends and leaving the means up for grabs (carpenter or product model)

22. 22 Standards-Based Reform The Initial Theory of Action Standards Assessment Accountability Clear Expectations Motivation Higher Student Learning Ala Tucker and Resnick in the early 1990s

23. 23 More Recent Expanded Theory of Action Standards Assessment Accountability Clear Exp’s Motivation Higher Student Learning Instruction Professional Development Ala Elmore and Resnick in the late 1990s.

24. 24 No Child Left Behind A definite retreat from performance- based assessment as a wide-scale tool A definite retreat from performance- based assessment as a wide-scale tool The persistence of standards-based reform... The persistence of standards-based reform... NCLB and Reading First NCLB and Reading First Every grade level outcomes assessment Every grade level outcomes assessment Assessments for placement, progress monitoring, diagnosis and outcomes Assessments for placement, progress monitoring, diagnosis and outcomes Mania for alignment… Mania for alignment…

25. 25 Alignment: Part of the theory of action 4 Specify the content standards 4 Align everything else to them: 4 Assessments 4 Instruction 4 Curriculum materials 4 Professional development 4 Standards for teacher learning

26. 26 Alignment: The centrality of content standards Content Standards Performance Standards For Students Student Assessments Wild Card Professional Development Standards for Teacher Education Accreditation Standards for Licensure Teacher Evaluation

27. 27 Problems in aligning assessments to standards The foxes are guarding the hen house problem (the test companies do the alignment) The foxes are guarding the hen house problem (the test companies do the alignment) The backwards mapping problem (When the standards and the instruction look just like the test): Deal with DIBELS later… The backwards mapping problem (When the standards and the instruction look just like the test): Deal with DIBELS later… The alignment is a coat of many colors problem (recall the earlier mapping) The alignment is a coat of many colors problem (recall the earlier mapping)

28. 28 Skill 1 Skill 2 Teach Assess Conclude Teach Assess Conclude The 1970s Skills management mentality: Teach a skill, assess it for mastery, reteach it if necessary, and then go onto the next skill. Remember the lessons of the 1970s!!! The bureaucratization problem

29. 29 Skill 1 Skill 2 Skill 3 Teach Assess Conclude Teach Assess Conclude Teach Assess Conclude Skill 4 Skill 5 Skill 6 Teach Assess Conclude Teach Assess Conclude Teach Assess Conclude The 1970s, cont. And we taught each of these skills until we had covered the entire curriculum for a grade level. The bureaucratization problem

30. 30 Standard 1 Standard 2 Teach Assess Conclude Teach Assess Conclude We could be back where we started. What could happen if we are not careful with standards The bureaucratization problem

31. 31 Standard 1 Standard 2 Standard 3 Teach Assess Conclude Teach Assess Conclude Teach Assess Conclude Standard 4 Standard 5 Standard 6 Teach Assess Conclude Teach Assess Conclude Teach Assess Conclude In the name of ensuring a place for each standard in our assessment systems, we could end up in the same place we were in in the 1970s What could happen if we are not careful with standards The bureaucratization problem

32. 32 DIBELS Detour Schooling the cognitive process of test taking: Kids who are alike in terms of accuracy will differ substantially in terms of speed and fluency Schooling the cognitive process of test taking: Kids who are alike in terms of accuracy will differ substantially in terms of speed and fluency Face validity problem: What virtue is there in doing things faster? Face validity problem: What virtue is there in doing things faster? naming letters, sounds, words, ideas naming letters, sounds, words, ideas What would you do differently if you knew that Susie was faster than Ted at X, Y, or Z??? What would you do differently if you knew that Susie was faster than Ted at X, Y, or Z???

33. 33 The Achilles Heel: Consequential Validity Give DIBELS Use results to craft instruction Give DIBELS again Give Comprehension Test The emperor has no clothes

34. 34 Key Concept: Haladyna Test Score Pollution: a rise or fall in a score on a test without an accompanying rise or fall in the cognitive or affective outcome allegedly measured by the test

35. 35 Rule of thumb re-emerges The finer the grain size at which we monitor a process like reading and writing, the greater the likelihood that we will end up teaching and testing bits and pieces rather than global processes like comprehension and composition. The finer the grain size at which we monitor a process like reading and writing, the greater the likelihood that we will end up teaching and testing bits and pieces rather than global processes like comprehension and composition.

36. Rule of Thumb for the Future Wherever we go, we must not go back to the world in which we link instruction to assessment at the very finest grain-size Wherever we go, we must not go back to the world in which we link instruction to assessment at the very finest grain-size Keep our eyes on the prize—what does accomplished performance look like? Keep our eyes on the prize—what does accomplished performance look like? 36

37. Where are we headed in the Post NCLB era? Obama administration keeping up the accountability pressure Obama administration keeping up the accountability pressure Teacher accountability for student achievement Teacher accountability for student achievement $350,000,000 investment in new accountability assessments (PARCC, SBAC) $350,000,000 investment in new accountability assessments (PARCC, SBAC) CCSS movement CCSS movement Another round of curriculum reform Another round of curriculum reform You are a part of that effort You are a part of that effort 37

38. A New Opportunity: A Confluence of Forces CCSS movements/Foundation funding CCSS movements/Foundation funding Disciplinary delivery of of literacy Disciplinary delivery of of literacy Reincarnation of Ambitious Pedagogy Reincarnation of Ambitious Pedagogy Deeper Learning Deeper Learning PARCC and SBAC PARCC and SBAC Hybrid assessments Hybrid assessments Increase weights given to open ended responses Increase weights given to open ended responses 38

39. Why performance assessment? 39

40. Performance Tasks: Why bother? External validity External validity College ready College ready Career ready Career ready PARCC ready PARCC ready Curricular validity Curricular validity Higher Order Thinking Higher Order Thinking Powerful learning Powerful learning Deeper learning Deeper learning

41. Performance Tasks: Why bother? Consequential validity Consequential validity What curricular activities will it lead teachers and students toward? What curricular activities will it lead teachers and students toward? Scaffolding toward independence Scaffolding toward independence Utility Utility Formative Assessment Formative Assessment Feedback to the student  differentiation Feedback to the student  differentiation Feedback to you as the teacher  reteaching Feedback to you as the teacher  reteaching

42. 42 Problems in aligning assessments to standards The assessment system problem The assessment system problem 4 One assessment will not do the job. 4 You need: 4 Big outcomes 4 Milestone assessments along the way (charting the trajectory of progress) 4 Analyses of the skill infrastructure 4 Matching tools to audience and purpose

43. PARCC digression 43

44. 44

45. 45

46. 46

47. My analysis Students who have learned how to read and write in response to your tasks Students who have learned how to read and write in response to your tasks AND AND Whatever curriculum it is that supports progress that will lead to good performance on those assessments Whatever curriculum it is that supports progress that will lead to good performance on those assessments Will do well on these PARCC tasks Will do well on these PARCC tasks 47

48. Why? They will have developed some transferrable practices that will serve them well in these new circumstances. They will have developed some transferrable practices that will serve them well in these new circumstances. That’s what we are about—developing transferable knowledge and skills That’s what we are about—developing transferable knowledge and skills 48

49. Closing Advice… Things you never really wanted to know or even ask about Things you never really wanted to know or even ask about Three things: Three things: Think about assessment as a system of indicators Think about assessment as a system of indicators Return to our accountability roots Return to our accountability roots Be careful about how we link instruction and assessment Be careful about how we link instruction and assessment 49

50. 50 Matching tools with decisions and clients Formative Assessments Assessment Systems

51. 51 Accountability Accountability follows responsibility Accountability follows responsibility Based upon multiple indicators (both external and internal), e.g., Based upon multiple indicators (both external and internal), e.g., External assessments (both state and standardized) External assessments (both state and standardized) Internal assessments (progress in our curriculum) Internal assessments (progress in our curriculum) Activity indicators (attendance, pass rates, library use) Activity indicators (attendance, pass rates, library use) Client satisfaction Client satisfaction Accountability should lead to assistance to build capacity Accountability should lead to assistance to build capacity Accountability should be reciprocal: Accountability should be reciprocal: Teachers take responsibility for learning Teachers take responsibility for learning Government for resources to support learning Government for resources to support learning Society for resources that lead to healthy lives Society for resources that lead to healthy lives

52. 52 Aligning everything to the standards: A model worth rejecting Standards AssessmentInstruction This model is likely to shape the instruction too narrowly. Lead to test score pollution.

53. 53 A better way of thinking about how standards can link instruction and assessment Standards our way of operationalizing our values: What we care about in teaching and learning literacy. Teaching and Learning Activities Assessment Activities They guide the development of both instruction and assessment This relationship can operate at the state or local level The logic of lots of good reform projects!

54. 54 My bottom line We desperately need instructionally sensitive assessments that have first rate psychometric characteristics so that we can build trustworthy internal systems for monitoring student progress We desperately need instructionally sensitive assessments that have first rate psychometric characteristics so that we can build trustworthy internal systems for monitoring student progress No decision of consequence about any individual, school, district or other aggregation should be based upon a single indicator of anything. No decision of consequence about any individual, school, district or other aggregation should be based upon a single indicator of anything. Tests are a means to an end: Their value is measured by the degree to which they allow us to make good decisions and provide good instruction. They are not the ends themselves. Tests are a means to an end: Their value is measured by the degree to which they allow us to make good decisions and provide good instruction. They are not the ends themselves. High stakes and low challenge is the worst possible scenario… High stakes and low challenge is the worst possible scenario…

55. 55 The real bottom line Never send a test out to do a curriculum’s job! Never send a test out to do a curriculum’s job! The End!!!! The End!!!! Instead, send the test out to support the curriculum and those held accountable to it