1. Interim Assessments: Do You Know What You Are Buying and Why? Scott Marion, Center for Assessment Imagine: Mathematics Assessment for Learning A Convening of Practitioners and Partners The Bill & Melinda Gates Foundation July 21, 2009

2. Marion. Center for Assessment. Gates Meeting 7/21/09 2 Our definition of interim assessment ► Assessments administered during instruction to evaluate students’ knowledge and skills relative to a specific set of academic goals in order to inform policymaker or educator decisions at the classroom, school, or district level. The specific interim assessment designs are driven by the purposes and intended uses, but the results of any interim assessment must be reported in a manner allowing aggregation across students, occasions, or concepts (Perie, Marion, & Gong, in press).

3. Marion. Center for Assessment. Gates Meeting 7/21/09 3 A definition of formative assessment ► Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes (FAST SCASS, 2006, emphases added).

4. Marion. Center for Assessment. Gates Meeting 7/21/09 4 Why are we here? ► We are working with the Foundation to develop a tool and supporting materials to assist educational leaders in selecting interim assessments for use in their states/districts/schools ► We have been conducting interviews with CAOs and principals to help us improve the evaluation tool ► We’d like to ask you a few of these key questions and gather your opinions and to test some ideas with you

5. Marion. Center for Assessment. Gates Meeting 7/21/09 5 Small Group Activity #1 ► Please describe in writing—on your own— the 2-3 major problems/issues that you are trying to address in your district, school, and/or classroom (up to 5 minutes)? ► Next discuss with your colleagues your answer and prepare to quickly report your answers (up to 5 minutes).

6. Marion. Center for Assessment. Gates Meeting 7/21/09 6 Small Group Activity #2 (15 minutes) ► If you have district responsibilities….  Are you considering or have you considered using an interim assessment help address any of these issues?  If not these issues, have you considered an interim assessment to address any specific issues in your district? Which issues? ► If you have school/classroom responsibilities…  Has your district required you to implement an interim assessment system?  If so, what was the stated reason(s)?  Are you using it for reasons other than the stated reasons?

7. Marion. Center for Assessment. Gates Meeting 7/21/09 7 Small Group Activity #3 (15 minutes) ► How do you (did you) envision the interim assessment helping to serve the “official” or “unofficial” purposes? ► What are the mechanisms by which these results/actions will occur?

8. Marion. Center for Assessment. Gates Meeting 7/21/09 8 Small Group Activity #4 (15 minutes) ► What features of your interim assessment system are critical to helping address these issues? ► What other features (currently not present in your system) would you like to see in an interim assessment for your district, school, and/or classroom? ► Did you look at the test items? If so, did you have content and assessment experts critically evaluate the items? What did you think?

9. Marion. Center for Assessment. Gates Meeting 7/21/09 9 Some thoughts about interim assessment ► Following are some of my thoughts about the features necessary for interim assessments to serve instructional purposes… ► After I present the next series of slides, I want you to challenge me and tell me where you think I’m incorrect (OK, you can also state your agreements too)

10. Marion. Center for Assessment. Gates Meeting 7/21/09 10 Theory of Action ► We have found it helpful to develop theories of action for a variety of assessment and accountability initiatives ► Trying to specify the mechanisms to explain why or how we think certain things will happen can reveal potential implementation gaps

11. Marion. Center for Assessment. Gates Meeting 7/21/09 11 A sample Theory of Action Interim test results Teachers /leaders meet to interpret results Leaders & teachers evaluate & modify programs Teachers determine and implement appropriate instructional strategies Student learning improves Interim test results improve Large scale test results improve

12. Marion. Center for Assessment. Gates Meeting 7/21/09 12 Mechanisms and criteria ► In order to get from one point to another within the circled boxes in the theory of action, we argue that users need to consider:  Timing (related to curriculum and instruction)  Content (focus, depth, and breadth)  Types of items/tasks (designed to provide summary information or insight into student learning)  Form of the results and feedback (summaries, descriptions)  Level of support

13. Marion. Center for Assessment. Gates Meeting 7/21/09 13 Timing ► Many argue that effective formative assessments must be intricately connected to the teaching and learning cycle to allow for frequent monitoring and adjustment (Sadler, 1989; Black & William, 1998; Shepard, 2000, 2007) ► Interim assessments that are administered only several times each year cannot meet the timing criterion  Is this a deal breaker for having interim assessments serve instructional purposes?  Must avoid Shepard’s “1000 mini-lessons” phenomenon  Interim assessments, however, might provide information useful for adjusting programs or targeting remediation efforts to specific groups of students

14. Marion. Center for Assessment. Gates Meeting 7/21/09 14 Items and Tasks ► Most formative assessment processes include a broad range of items, questions, observations, and extended tasks  These “observations” tend to be at a pretty fine-grained level since they are often “in the minute” ► Most commercial interim assessment systems rely almost exclusively on multiple-choice or other selected response formats  Collection of items tend to be a sampling of the domain and do not typically provide the fine-grained information available with formative assessments  In the worst case, we have seen many collections of lousy items

15. Marion. Center for Assessment. Gates Meeting 7/21/09 15 Nature of content ► In order to serve instructional purposes, the content assessed must be closely related to curriculum (not just the standards) in which the students are engaged ► The specific knowledge and skills targeted should be—to the extent possible—tied to a learning progression that illustrates how students will develop proficiency in the domain ► The content of the assessment should represent the standards at least at depth of knowledge called for in the standards ► Most interim assessments that we have seen are tied to most generic descriptions of content rather than what might be necessary for curriculum- specific feedback

16. Marion. Center for Assessment. Gates Meeting 7/21/09 16 Nature of Results/Reports/Feedback ► Interim assessment results are aggregated beyond single classrooms  Typically involves some type of data reduction  Growing body of work about effective score reporting can be utilized to design reports as instructionally useful as possible  Types of available data summaries can provide useful evaluative information, but not necessarily instructional ► We argue that for interim assessments to provide instructionally useful information, qualitative information must be included on score reports  Information on correct/incorrect responses by sub- domain  Feedback on what an incorrect answer implies  Suggestions for next steps

17. Marion. Center for Assessment. Gates Meeting 7/21/09 17 Levels of support ► If teachers could do this already, we wouldn’t be here ► It is crucial to figure out how to support teachers as they take data and information and turn it into decisions and actions  Models  Mentoring  On-going training ► Teachers receive these types of supports when they engage in high-quality professional development to learn formative assessment processes

18. Marion. Center for Assessment. Gates Meeting 7/21/09 18 Checking In ► So, can interim assessments serve instructional purposes?  What features do you think are absolutely necessary? ► Have you seen evidence that the assessments you are using have led to improvements in teaching and learning?  How and why have these results occurred?

19. Marion. Center for Assessment. Gates Meeting 7/21/09 19 References ► Perie, Marion, & Gong, in press EM:IP ► Perie, Marion, Gong, & Wurtzel, 2007, posted on the CCII, NCIEA, and Achieve websites) ► For more information:  www.nciea.org www.nciea.org  smarion@nciea.org smarion@nciea.org