1. Copyright © 2007 Educational Testing Service Integrating learning theory in an era of accountability testing Drew Gitomer ETS Conference on Educational Testing in America: State Assessment, Achievement Gaps, Federal Policy and Innovations September 8, 2008

2. 2 The Problem We Are Confronting Accountability assessments grounded in an outdated scientific model for conceptualizing proficiency, teaching it, and assessing it Accountability is based on a limited set of limited proxies Interim assessments, formative assessments, and teacher professional development that are emulating the less desirable characteristics of those accountability tests (aka the irony of alignment)

3. 3 The Vision Can have an assessment system that contributes to educational practice by: Documenting what students have achieved (“assessment of learning”), Helping identify how to adjust instruction (“assessment for learning”), and Engaging students and teachers in worthwhile educational experiences in and of themselves (“assessment as learning”)

4. 4 Coherence of Three Components Public accountability functions Formative assessment Professional development What is coherence?

5. 5 Current Focus of the Project Reading, writing and mathematics Middle school Primary work in Maine (Portland) Beginning work in other places Today’s talk focuses on reading and the work cited is led by Kathy Sheehan and Tenaha O’Reilly

6. 6 The Foundation: A Strong Conceptual Base Built on: –Cognitive-scientific research Principles Domain-specific competency models Developmental models as available –Aligned with State standards

7. 7 The Reading Competency Model: The Skills Branch Understand & Use Texts Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Pre-Reading Strategies Model Building Strategies Strategies for Going Beyond the Text Required Skills Reading Strategies Text Conv. & Characteristics Persuasive Text Informational Text Literary Text Learning to Read Reading to Learn Reading to Do

8. 8 The Reading Competency Model: The Strategies Branch Understand & Use Texts Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Pre-Reading Strategies Model Building Strategies Strategies for Going Beyond the Text Required Skills Reading Strategies Text Conv. & Characteristics Persuasive Text Informational Text Literary Text Deliberate, conscious, effortful actions that readers can implement to repair breaks in comprehension & enhance understanding. Ask/answer pre-reading questions Use Graphic Organizers Adopt a Critical Stance

9. 9 The Reading Competency Model: The Text Branch Read Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Pre-Reading Strategies Model Building Strategies Strategies for Going Beyond the Text Required Skills Reading Strategies Text Conv. & Characteristics Persuasive Text Informational Text Literary Text Knowledge of conventions that contribute to reading ease or difficulty

10. 10 The Text Branch Difficulty Drivers The Text Genre Informational Research/Persuasive LiteraryVocabularySyn. ComplexityCohesion Structure, Negation, etc. (Based on Sheehan, Kostin & Futagi, in press) Ensure that: (1) targeted skills are assessed within a grade- appropriate span of text complexity; and (2) assessment tasks focus on those aspects of text variation that are likely to have a significant impact on reading performance.

11. 11 The Skills Tested by Most Existing Reading Comprehension Tests Recognize Common Words Integrate Inf w Bk. Know. Integrate Inf. Across Texts Evaluate/Critique Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Decode Unfamiliar Words Read Fluently Locate/Retrieve Inf. Und. Diff. Voc./Lit. Term. Infer Relations Amg Concepts Und. & Use Discourse Struc. Understand & Use Texts

12. 12 Key Features - Reading Design Measure the full range of skills, including prerequisite skills Purposeful, authentic, scenario-based tasks Support reading across the curriculum –Informational Texts: Excerpts from the introductory chapters of science and social studies textbooks, newspaper & magazine articles, etc. –Persuasive Texts: Editorials, Letters to the editor –Literary Texts: Narratives, memoirs, poems Select task formats that encourage use of targeted reading strategies Include a mixture of expressive (CR) and receptive (mc) tasks Minimize production requirements –Want to test reading not writing

13. 13 Implementation: The Assessment Design Period accountability assessments (PAAs) Foundational tasks –Constructed response and selected response –Scaffolded structure Supplemental tasks Computer Administration Automated Scoring

14. 14 CBAL vs. (stereo)typical assessments TraditionalCBAL Single Measurement Occasion Multiple Measurement Occasions Many short itemsA few long tasks (mostly) unrelatedCentered around a common theme Representative of a domain Based on a Competency Model Homogeneous Response Types Heterogeneous Response Types

15. 15 A Sample PAA Spoken Module –Provide mastery evidence relative to the Prerequisite Skills –Short Tasks: Word Recognition, Decoding, Oral Fluency –Students speak into a headset, responses are scored as Words Correct per Minute (wcpm) Comprehension Module –Provide mastery evidence relative to the Model-Building & Applied Comprehension Skills –All texts & tasks revolve around a meaningful scenario e.g., You have to write a report about the scientific method for your science class –Tasks employ innovative response formats: SR and CR –Multi-step thinking, scored via partial credit scoring rubrics

16. 16 The Scenario Four Texts –A passage about Ben Franklin adapted from a Social Studies textbook –An article about the Scientific Method extracted from an Encyclopedia –A newspaper article about 3 winners of the Intel Science Competition –A diagram from a student lab report You have to write a report about the Scientific Method for your science class. Since you enjoy reading about American History, you decide to focus your report on Ben Franklin’s use of the Scientific Method. Read the following passages to learn more about the Scientific Method and about Ben Franklin’s scientific experiments.

17. 17 Preliminary Results – Construct Validation TOWREGatesz’ The Spoken Module 0.78 0.63 2.69 ** The Comprehension Module 0.46 0.79 5.18 *** TOWRE: a standardized measure of Prerequisite Skills Gates-MacGinitie: a std. measure of Model-Building Skills

18. 18 The Assessment is Also Designed to Yield Diagnostic Information that Teachers can Use to Adjust Instruction

19. 19 Formative System Is built upon the same conceptual base as the Accountability System Gets information from the Accountability System Is used at the district’s and teacher’s option Is available on demand Performance in the formative system will not be used for accountability purposes

20. 20 Structure of formative assessment Library of resources including professional development component –Elaborated foundational tasks – for each task component Clarification of learning intentions – connection to competency model Criteria for considering student responses Common conceptual challenges Potential pedagogical strategies Potential additional resources

21. 21 Challenges Psychometric Automatic scoring Educational infrastructure Political accountability