1. Fidelity of Implementation A tool designed to provide descriptions of facets of a coherent whole school literacy initiative. A tool designed to provide reflection opportunities that result in ongoing progress of student achievement over time. A tool designed to help literacy teams establish and prioritize goals for the upcoming year.

2. Fidelity of Implementation Package Fidelity of Implementation Tool Fidelity of Implementation Achievement Data Fidelity of Implementation Teacher Surveys

3. Developing an Evaluation Plan A PowerPoint program evaluation tool for Literacy Collaborative Schools (Revised 3/12)

4. Developing an Evaluation Plan This presentation is intended to guide school-based literacy teams through the process of developing a school- or district-level evaluation plan. It is divided into three sections, which can be used separately or together.

5. Developing an Evaluation Plan The three sections correspond to the three goals of the presentation: Goal 1: To understand the purpose of the evaluation plan and the process for developing it Goal 2: To formulate evaluation questions and to develop a plan to answer those questions Goal 3: To put the plan into action

6. As a team... Reflect on the following slides Share your thinking Ask questions Have a scribe take notes Make decisions Formulate evaluation questions and develop a school- or district-level evaluation plan

7. Goal 1 To understand the purpose of the evaluation plan and the process for developing it.

8. Purpose of the Evaluation Plan All Literacy Collaborative schools develop a school- or district-level evaluation plan to track student progress and evaluate the effectiveness of their Literacy Collaborative implementation over time.

9. How the process works...

10. Fidelity of Implementation Timeline – Getting Started Literacy Team begins using the Fidelity of Implementation Tool to describe Successes, Challenges, and Action Plans. Mid- Year Literacy Team develops an evaluation plan and submits it to the training site by the end of September. Spring/Summer * This date may differ for district sites. Check with your trainer.

11. Fidelity of Implementation Future Timeline Literacy team reviews evaluation questions & plan, modifies if necessary based on previous year’s data results (beyond field year), and plans for current year’s data collection Data are collected based on the evaluation plan Fall During the year Spring/Summer Data are gathered, reviewed and analyzed by the team and presented to the university site.

12. state the questions that have been identified by the team, specify the assessments that the school will use for the next five years, specify when the assessments will be administered, include plans for analyzing and presenting results. The evaluation plan will...

13. …be based on the information your school or district has to develop for other plans (Title 1 plans, School Improvement Plans) so you do not have to duplicate your efforts. The evaluation plan can...

14. The evaluation plan can… …include measures of teacher change as well as student-level measures of achievement. These could include teacher surveys or interviews that demonstrate positive changes in teacher practice or understanding over time.

15. Fidelity of Implementation: Teacher Surveys Two Kinds of Teacher Survey: Implementation Year Ongoing Implementation

16. Fidelity of Implementation: Teacher Surveys Surveys are divided into three parts: Implementation Professional Development Coaching

17. ..include results of some of the required classroom assessments listed in the Literacy Collaborative Classroom Assessment Charts (www.literacycollaborative.org). You may also use alternative assessments from the BAS-1.www.literacycollaborative.org The information gathered from classroom assessments is designed to be used primarily to inform classroom instruction, but results from some of these assessments can be used to answer program evaluation questions as well. The evaluation plan can...

18. For example, Benchmark Assessment System… CAN BE USED TO EVALUATE PROGRAM EFFECTIVENESS if given in the fall and again at the end of the year to assess student progress in reading. Can be reported across an entire grade level to demonstrate changes in the number of students performing below, at, and above instructional level for each grade. CAN BE USED TO INFORM INSTRUCTION if used at the beginning of the year to determine instructional level and placement in guided reading groups.

19. Example of an assessment used to inform classroom instruction Running Records Used twice a month to determine a student’s reading level, accuracy, use of sources of information, and reading behaviors. Teachers use this information to assist with group placement and individual instruction.

20. Make a List of Your Assessments Make a list of all of the literacy assessments you currently administer. Make sure you have listed all of the reading, writing, and word-study assessments at each grade level. What information does each of these provide (e.g., area tested, type of score)? Which of these assessments can be used to track change over a year, multiple years? Identify any assessments that duplicate the kind of information you are looking for and choose which one to include.

21. Your list could include: Literacy Collaborative Required Classroom Assessments – Reading – Writing – Phonics/Word-Study Standardized Tests – State Mandated Grade Level Assessments – TerraNova, Gates-MacGinite, Stanford9, CAT5 District or Other Required Assessments – BAS -1 Other – student attitude surveys – teacher surveys

22. Goal 2 To formulate your evaluation questions and develop a plan to answer those questions.

23. The examples on the following pages may help you think about the kinds of questions you might want to ask in the areas of reading, writing, and word-study. Read the examples and discuss as a team possible evaluation questions for your school. As you consider possible questions, think of your stakeholders and what they might want or need to know. Developing Questions Activity

24. Possible Evaluation Questions (cont’d) What percentage of third grade students is scoring proficient or above on our state reading assessment each year? Are more students scoring proficient or above in reading as we implement Literacy Collaborative?

25. Possible Evaluation Questions (cont’d) What is the average HRSIW score for first- grade at the beginning of each year? Across years? What is the average HRSIW score for each class of incoming kindergartners in the fall? Is the kindergarten teaching making a difference for first-graders as measured by changes in their average HRSIW scores in the fall over time?

26. Formulate your questions List the questions your team decides to pursue. Remember to make your questions clear, focused and measurable. Your questions may focus on one year only (fall to spring) or may focus on multiple years.

27. Revisit your list of assessments Determine which assessments will allow you to answer your evaluation questions. If you are not currently administering assessments that will allow you to answer your evaluation questions, identify assessments that will allow you to do so and add them to your list.

28. Write up your Evaluation Plan Using all of the information that you have developed (your evaluation questions and your list of assessments), you can now write up your evaluation plan. You can do it in chart, list, or table format.

29. Sample Evaluation Plans

32. Making Sense of Quantitative Student Data Thinking about numbers takes mental work, is not easy and is not an “objective” process. Numbers have to be organized and summarized in some way in order to think about them. Numbers cannot communicate on their own. Discussing and writing about numbers is a way to find meaning in numbers.

33. There are many ways of analyzing the same data, and the choices you make depend on your questions and interests. There is no “right” way and “wrong” way. Which is why one person shouldn’t do it alone: analyzing data should involve multiple perspectives.

35. “Average word reading scores increased substantially over the year, from 6.3 in the fall to 11.3 in the winter to 16.5 in the spring.”

36. With all graphs, be mindful of scale. It’s usually best (though not always) to use the full scale of a measure when graphing it.

39. Summarizing results from your data analyses Summaries should be short so that someone who isn’t going to read your whole report will be able to quickly get an overview of student learning. (1-2 paragraphs) Generalize across grade levels, across assessments, across the year, and across years (if you have multi-year data). Include qualitative information about student learning as well as quantitative.

40. Summarizing Data Analyses: Example “Overall, the K-2 students did well this year, with teachers reporting that students were enthusiastic about the literacy activities….etc. While only 40% of entering kindergarteners were reading on grade level according to the BAS, 70% were by the end of the year. Similarly, 1 st and 2 nd graders showed increases in the percentages of students on grade level, from roughly 50% in both grades in the fall to 75% of first graders and 60% of second graders in the spring. First grade scores were higher than last year’s first grade scores, but 2 nd grade scores were slightly lower than last year’s 2 nd grade scores. Most special needs students in all three grade levels made small gains in reading. In all three grades students showed particular strength in writing…”

41. Goal 3 To put your plan into action.

42. Your questions How you will answer them (which assessments you will administer at different points during the year) How you will analyze your data Your evaluation plan consists of…

43. To put your plan into action, you will need to decide… Who will be a part of the process? What will happen? When it will happen? What you will do with the information?

44. Make a Timeline When will the assessments be administered? Who will be responsible for administering, collecting, and analyzing the data? When will the team gather to examine the data and talk about the results? What needs to be done to facilitate this process?

45. Congratulations! You have completed the first important steps toward tracking your students’ progress and evaluating the effectiveness of your Literacy Collaborative implementation over time.