Slide 1, Moving From Challenge To Action: Accountability Supporting Student Learning Joan L. Herman UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) CRESST Conference UCLA, Los Angeles, CA September 9, 2004

Slide 2, Overview Role of assessment in improving student learning Evidence on how things are working Advice on moving ahead

Slide 3, Theory of Action: Motivation Establish standards Develop measures Set performance goals Leaven with incentives/sanctions Schools will be pay attention Attend to standards/assessment results Work to improve student performance/learning

Slide 4, Theory of Action: Technical System Assessment results will provide accurate/valid information at multiple levels Data will be well used to inform planning and decision making Educational system will well use data to engage in continuous improvement

Slide 5, Is the Motivation System Working? The good news: Districts and schools taking action to align curriculum, etc with standards Teachers listen to the signal and focus instruction accordingly The not so good news Curriculum narrowed Possibility of dual curriculum Concern for teacher’/principals’ professionalism and morale

Slide 6, Some Critical Questions: How can we design standards and assessment systems that optimally focus instruction but don’t stifle it? What happens to motivation when schools hit the wall and performance levels out?

Slide 7, Is the Technical System Working? Feds are asking for right kinds of validity evidence Alignment Validity for EL and SWD Accuracy: 75% probability that CST correctly classifies students’ proficiency level (Rogosa) Reliability and year to year score fluctuations a continuing challenge

Slide 8, Is the Technical System Working? (cont.) Validity of gains suspect Feasibility of AYP targets problematic Thorny technical/psychometric issues in dealing with 40+ possible indicators of AYP

Slide 9, Critical Questions Validity of AYP designations: Do they identify the right schools? Is there a better way to establish and operationalize improvement targets?

Slide 10, Alignment: A Critical Lynchpin Are we clear enough about what we want: Teachers to teach? Test developers to test? Is face validity of content and cognitive demand enough? Research suggests difficulties

Slide 11, One Alignment Example Study used 20 content experts Considered each item and rated Topic addressed Depth of knowledge (DOK) Content centrality Results - Clear majority agreed on Topic for 35/42 items Topic AND DOK for 14/42 items Topic, DOK and centrality for 5 items

Slide 12, Implications How can teachers teach to “standards” if experts don’t agree on what topics and levels of DOK mean? Current reality: alignment is a moving target

Slide 13, A Moving Target Example from NY (Alan Tucker) Standard: Apply formulas to find measures such as length, area, volume, weight, time, and angle in real world situations Representation of Math A Test, June, 02 - June, 03 (4 tests) # items: 1-3 Range of content and complexity

Slide 14, Critical Questions If we want teachers to teach to the standards rather than the test, how can we be sure that developers and teachers share the same understandings? How can we specify standards and assessments in ways that clearly communicate expectations without unacceptable curriculum narrowing? Can we frame expectations in ways that are feasible for all students?

Slide 15, Teacher Assessment: The Neglected Key If we believe in the power of assessment to support learning, we must put that power where learning occurs Many capacity challenges Some realistic remedies: Teacher pre-service education Requirements for materials developers

Slide 16, A Full Agenda for Deliberation and Action