1. From Evidence to Action: A Seamless Process in Formative Assessment? Margaret Heritage Jinok Kim Terry Vendlinski American Educational Research Association Annual Meeting New York, NY - March 24-28, 2008

2. Evidence to Action: A Seamless Process? A Generalizability study of measures of teacher knowledge Teachers can draw appropriate inferences from evidence Difficulties in deciding next instructional steps

3. Formative Assessment

4. An ongoing process to gather evidence and provide feedback about learning Use the feedback to identify the gap between current learning and desired goals (Sadler, 1989)

6. Teacher Knowledge Measures

7. Measures of Teacher Knowledge for Teaching Mathematics Drawn from POWERSOURCE Conceptualize teacher knowledge as central to and embedded in the everyday practice of teaching Key principles underlying mastery of algebra I Distributive property Rational number equivalence Solving equations

8. Student Assessment

12. Teacher Knowledge Measures 1. What is the key principle that these assessments address? Why do students need to understand this principle for algebra I? 2. What inferences would you draw from this student’s responses? What does this student know? What does this student not know? 3. If this student were in your class, based on your responses to questions 2 and 3, what would you do next in your instruction? 4. If you were this child’s teacher, what written feedback would you give to this student?

13. Scoring Rubric: Next Instructional Steps for Teaching the Distributive Property ScoreCriteria 4  Explain the distributive property as repeated addition  Explain factoring as distribution in reverse  Model the use of the distributive property with whole numbers  Model generalizing to other numbers and variables 3Either  Explain the distributive property as repeated addition Or  Explain factoring as distribution in reverse  Model the use of the distributive property with at least whole number 2  Explain procedures for how to use the distributive property, equating procedures with the order of operations 1  No explanation of the distributive property

14. Generalizability (G) Study

15. G-Theory Framework Object of Measurement: Teachers’ pedagogical knowledge in mathematics Aims: To assess the generalizability of a task score in a single condition to an average score in a countless combination of conditions Determine the source of error if the task score is not generalizable

16. Initial Assumptions: 3 Potential Sources of Measurement Error Variations in raters’ scoring of a teacher’s responses A teacher’s knowledge of principles may vary A teacher’s score may vary depending on the task

17. Sample of Participating Teachers 118 6th grade teachers from a variety of districts in Los Angeles County 90% credentialed for K-6, 60% for 7-8, 34% grade 9, 20% for 10-12 90% general credential 14% mathematics credential

18. Analysis o xr x p x to xr x p x t object of measurement (teacher) raterprincipletype of task

19. Variability Findings: Main Effects Main effect of raters = 0.2% The impact of raters is negligible Main effect of type of task = 25% Teachers’ scores may not be generalizable across tasks

20. Variability Findings: Interactions Interactions of object of measurement (teacher), principle, and task = 39.4% Some principle-task combinations are more difficult for some teachers than for other teachers Interactions of principle and task = 8.2% Some combinations of principle and task more difficult

21. VariableNMeanStd DevMinMax Principle: distributive property Task: identifying key principle1142.070.6313.67 Task: evaluating student understanding 1132.140.9414.00 Task: planning next instruction1131.21  0.3612.00 Principle: solving equations Task: identifying key principle1121.820.8913.83 Task: evaluating student understanding 1112.060.9313.83 Task: planning next instruction1121.21  0.3912.83 Principle: rational number equivalence Task: identifying key principle 1052.940.4714.00 Task: evaluating student understanding 1022.070.9814.00 Task: planning next instruction1011.36  0.4812.83

22. Improving the Translation of Evidence to Action

23. Know Which Way to Go?

25. Learning from Teaching

26. And Finally … Challenge of developing valid and reliable measures of teacher knowledge Using assessment information to plan subsequent instruction tends to be the most difficult task Evidence may provide the basis for action but cannot itself “form” the action

27. mheritag@ucla.edu