1. Advancing Assessment Literacy Data Gathering I: Establishing Outcomes

2. Data Gathering I: Establishing Outcomes Part One: Beginning the Search

3. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 3 Rearview Mirror Effect Planning for the future on the basis of events past is the school planning problem Stephen White (2005) identified as the Rearview Mirror Effect. White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

4. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 4 Four Debilitating Characteristics of the Rearview Mirror Effect 1.Responding to a rapidly changing reality based on past events. Non-anticipatory Doesn’t seek information about the current reality from students and parents This is the wait and see approach to education – “We’ll see if this literacy thing works before we adopt it for ourselves”. White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

5. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 5 Four Debilitating Characteristics of the Rearview Mirror Effect 1.Responding to a rapidly changing reality based on past events. What do you anticipate will need to be addressed in the future regarding student learning? How much do you know about the current reality of students, parents and teachers? White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

6. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 6 Four Debilitating Characteristics of the Rearview Mirror Effect 2.Waiting for the road to reveal itself, by depending on and waiting for biannual assessments. This is a common concern regarding the Assessment for Learning results – the data comes too late to matter and the decisions we make don’t affect the students who were tested. White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

7. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 7 Four Debilitating Characteristics of the Rearview Mirror Effect 2.Waiting for the road to reveal itself, by depending on and waiting for annual or biannual assessments. In what ways might the road be made more clear between annual or biannual assessments? What is known about the function of assessments such as Assessment for Learning and CAT3? White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

8. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 8 Four Debilitating Characteristics of the Rearview Mirror Effect 3.Focusing on a single dimension of the highway; that is, a focus only on what students do. We must look at the interrelationship inherent within problems, processes and systems. We must look at data from a wide variety of sources to inform planning. White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

9. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 9 Four Debilitating Characteristics of the Rearview Mirror Effect 3.Focusing on a single dimension of the highway; that is, a focus only on what students do. What other types of data might need to be gathered and looked at to paint a more detailed picture of student achievement? White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

10. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 10 Four Debilitating Characteristics of the Rearview Mirror Effect 4.A wistful looking back to a time when things were simpler. “I wish I could go back to just teaching, rather than constantly assessing.” White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

11. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 11 Four Debilitating Characteristics of the Rearview Mirror Effect 4.A wistful looking back to a time when things were simpler. Is teaching today the same as at the beginning of your career? What changes have you noticed? What things have improved? Stayed the same? Deteriorated? White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.

12. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 12 Beginning the Search: Looking at Archival Data You have been supplied with three or more years of archival data. Example: Course Averages by Grade – final grades out of a possible 100. Grade 7200420052006 ELA787475 Math758068

13. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 13 Beginning the Search: Looking at Archival Data What patterns or trends seem to be emerging from the data? Craft general statements based on what you see; for example: – Student performance in English seems to be consistent. – Math performance was increasing but has decreased significantly since 2005.

14. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 14 Beginning the Search: Affinity Diagramming As a table group, choose one of the statements that you believe is most pressing. Brainstorm hunches and hypotheses that might explain why this situation exists. –Put each on a sticky note. (Avoid one word responses.) –Brainstorm at least 20 hunches/hypotheses. –For example, “Math achievement is low because teachers are struggling with the format of the curriculum guide.” Bernhardt, V. L. (2004). Data analysis for continuous school improvement. Larchmont, NY: Eye on Education.

15. 15 Beginning the Search: Affinity Diagramming When you are done brainstorming: –Look over the sticky notes and sort the items into categories. –Create a title for each category. Title Bernhardt, V. L. (2004). Data analysis for continuous school improvement. Larchmont, NY: Eye on Education.

16. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 16 Beginning the Search: Affinity Diagramming This process allows for the identification of a variety of seemingly unrelated reasons and can lead to the revelation of interesting interrelationships. From your Affinity Diagram, select the category or categories that contain the majority of hunches and hypotheses. Bernhardt, V. L. (2004). Data analysis for continuous school improvement. Larchmont, NY: Eye on Education.

17. 17 Identifying Effects and Drivers: Interrelationship Diagramming Within each category it is important to separate drivers (causes) from effects. –For example, introduction of a breakfast program (driver) will improve student concentration (effect). Wellman, B. & Lipton, L. (2004). Data-driven dialogue. Mira Via, LLC. Arrows moving away from boxes indicate drivers, while arrows moving into boxes indicate effects. A sample interrelationship diagram is on the following slide. Breakfast Program Student Concentration Driver Effect

18. Wellman, B. & Lipton, L. (2004). Data-driven dialogue. Mira Via, LLC. Division Division Standards

19. 19 Creating an Interrelationship Diagram Take the sticky notes from the longest list and place them in a circle with the topic as a header above: TOPIC (The title of the category) Wellman, B. & Lipton, L. (2004). Data-driven dialogue. Mira Via, LLC.

20. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 20 Creating an Interrelationship Diagram Select one item in the category as a starting point. Ask two-way questions to determine whether this item is a driver or effect of each of the other items. For example, “Is provision of a breakfast program a driver of student concentration or is student concentration a driver of providing a breakfast program?” Draw arrows from the drivers to the effects. Continue in this way through each of the items. NO two-headed arrows! Decide which item dominates the other. Wellman, B. & Lipton, L. (2004). Data-driven dialogue. Mira Via, LLC.

21. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 21 Creating an Interrelationship Diagram DRIVERS –Drivers are indicated by the number of arrows going away from an item. –Count the number of arrows going away from each item. –Rank the drivers from highest to lowest. EFFECTS –Effects are indicated by the number of arrows pointing towards an item. –Count the number of arrows pointing towards each item. –Rank the effects from highest to lowest. Wellman, B. & Lipton, L. (2004). Data-driven dialogue. Mira Via, LLC.

22. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 22 Creating Hypotheses Based on Drivers Select the top 3-5 drivers. Create a hypothesis for each. For example, –“Math achievement dropped because teachers are struggling with the format of the curriculum guide.” –“Math achievement has been decreasing because their background understanding is insufficient.”

23. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 23 Testing Validity The hypotheses that have been generated are based on the experiences and perceptions of the group. Before committing to action regarding these hypotheses, it is necessary to further check their validity to ensure that they are truly drivers of student learning in the areas you have identified. Literature regarding these hypotheses will be gathered and shared at your next meeting.

24. Data Gathering I: Establishing Outcomes Part Two: Testing the Validity of Hypotheses

25. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 25 Testing the Validity of Hypotheses Hypotheses were created based on the drivers identified at our last meeting. Today, we are going to check the validity of those hypotheses against current literature in those areas.

26. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 26 Testing the Validity of Hypotheses Divide into equally sized groups based on the number of hypotheses to be tested. Each group will receive the hypothesis and one or more current articles pertaining to that hypothesis.

27. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 27 Testing the Validity of Hypotheses The task: Read the articles related to the hypothesis you’ve been assigned. How is this hypothesis reflected in the literature? What key elements are present or missing? Is the hypothesis supported by the literature? (i.e. Is it really an issue?) How might the hypothesis need to be changed to reflect current research in this area? Is this hypothesis valid? Is it truly a driver of student learning in this area? Be prepared to present your findings to the larger group.

28. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 28 Presentations Read your hypothesis to the group. Present the findings from your reading. Give your recommendation regarding the validity of this hypothesis. Once all have been presented, decide which hypotheses you would like to explore.

29. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 29 Converting Hypotheses to Goal Statements Once hypotheses have been identified as valid and worthy of pursuit, they must now be converted into goal statements. For example: –“Math achievement is low because teachers are struggling with the format of the curriculum guide.” will now be converted to: –“Math teachers will be supported in their understanding and actualization of the curriculum guide in order to improve student learning.”

30. Advancing Assessment Literacy Modules: Data Gathering I - February 2008 30 Identified Outcomes The goal statements that you have created are now the identified general outcomes for your staff and students. These statements are broad and will require further unpacking to flesh out the indicators of success as you work toward achieving your goals. At a later date, these general goal statements will serve as the foundation of a goal statement such as SMART goals.