1. Slide 1, 9-9-04 Moving From Challenge To Action: Accountability Supporting Student Learning Joan L. Herman UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) CRESST Conference UCLA, Los Angeles, CA September 9, 2004

2. Slide 2, 9-9-04 Overview Role of assessment in improving student learning Evidence on how things are working Advice on moving ahead

3. Slide 3, 9-9-04 Theory of Action: Motivation Establish standards Develop measures Set performance goals Leaven with incentives/sanctions Schools will be pay attention Attend to standards/assessment results Work to improve student performance/learning

4. Slide 4, 9-9-04 Theory of Action: Technical System Assessment results will provide accurate/valid information at multiple levels Data will be well used to inform planning and decision making Educational system will well use data to engage in continuous improvement

5. Slide 5, 9-9-04 Is the Motivation System Working? The good news: Districts and schools taking action to align curriculum, etc with standards Teachers listen to the signal and focus instruction accordingly The not so good news Curriculum narrowed Possibility of dual curriculum Concern for teacher’/principals’ professionalism and morale

6. Slide 6, 9-9-04 Some Critical Questions: How can we design standards and assessment systems that optimally focus instruction but don’t stifle it? What happens to motivation when schools hit the wall and performance levels out?

7. Slide 7, 9-9-04 Is the Technical System Working? Feds are asking for right kinds of validity evidence Alignment Validity for EL and SWD Accuracy: 75% probability that CST correctly classifies students’ proficiency level (Rogosa) Reliability and year to year score fluctuations a continuing challenge

8. Slide 8, 9-9-04 Is the Technical System Working? (cont.) Validity of gains suspect Feasibility of AYP targets problematic Thorny technical/psychometric issues in dealing with 40+ possible indicators of AYP

9. Slide 9, 9-9-04 Critical Questions Validity of AYP designations: Do they identify the right schools? Is there a better way to establish and operationalize improvement targets?

10. Slide 10, 9-9-04 Alignment: A Critical Lynchpin Are we clear enough about what we want: Teachers to teach? Test developers to test? Is face validity of content and cognitive demand enough? Research suggests difficulties

11. Slide 11, 9-9-04 One Alignment Example Study used 20 content experts Considered each item and rated Topic addressed Depth of knowledge (DOK) Content centrality Results - Clear majority agreed on Topic for 35/42 items Topic AND DOK for 14/42 items Topic, DOK and centrality for 5 items

12. Slide 12, 9-9-04 Implications How can teachers teach to “standards” if experts don’t agree on what topics and levels of DOK mean? Current reality: alignment is a moving target

13. Slide 13, 9-9-04 A Moving Target Example from NY (Alan Tucker) Standard: Apply formulas to find measures such as length, area, volume, weight, time, and angle in real world situations Representation of Math A Test, June, 02 - June, 03 (4 tests) # items: 1-3 Range of content and complexity

14. Slide 14, 9-9-04 Critical Questions If we want teachers to teach to the standards rather than the test, how can we be sure that developers and teachers share the same understandings? How can we specify standards and assessments in ways that clearly communicate expectations without unacceptable curriculum narrowing? Can we frame expectations in ways that are feasible for all students?

15. Slide 15, 9-9-04 Teacher Assessment: The Neglected Key If we believe in the power of assessment to support learning, we must put that power where learning occurs Many capacity challenges Some realistic remedies: Teacher pre-service education Requirements for materials developers

16. Slide 16, 9-9-04 A Full Agenda for Deliberation and Action