1. Linking a Comprehensive Professional Development Literacy Program to Student Achievement Edmonds School District WERA December 4, 2008

2. Presenters Nancy Katims, Director of Assessment, Research, and Evaluation N. Lynn Caulkins, Assessment and Data Specialist Maggie Conners, District K-6 Literacy Coach Lara Drew, Director of Elementary Education

3. Purpose of Study To determine whether the set of professional development activities provided through the Collaborative Literacy Project (CLP) has a measurable impact on student performance in literacy.

4. Purpose of this Presentation To describe an evaluation design that we found helpful in investigating the effectiveness of a comprehensive professional development program.

5. Agenda Introduction Purpose Description of the Collaborative Literacy Project (CLP) Description of the Evaluation Design Findings of Two Studies Evaluating the Design Next Steps

6. What is the Collaborative Literacy Project?  Professional development model designed to improve student learning in reading and writing in grades K-6.  Provides in-depth training in literacy for principals and teachers.  Develops demonstration classrooms and learning labs across the district.  Develops a literacy coaching model.

7. What is the Collaborative Literacy Project?  Learning is focused on effective instructional practices in literacy.  Teaching for Understanding  Explicit Instruction of Metacognitive Strategies  Using the Gradual Release of Responsibility  Using a Workshop Structure

8. What is the Collaborative Literacy Project?  Developed literacy coach model to support school-based professional learning in literacy.  Provide one-on-one coaching  Support small group learning opportunities  Facilitate whole staff learning opportunities  Facilitate classroom observations  Support leadership of principals and teacher leaders  Collect and share resources

9. Difficulties of Study CLP encompasses many types of professional development (PD). Teachers participate in a variety of activities that overlap. How do we isolate the effects of the CLP?

10. Study 1 Student Achievement in the 06-07 School Year

11. Evaluation Design Listed the major CLP PD activities. Gave a value weight to each activity (see page 1 of report). Tallied the number of activities in which each K-6 teacher participated in 05-06 and 06-07. Assigned “CLP PD Points” to each elementary teacher.

12. Evaluation Design Our K-6 teachers had CLP PD points that ranged from zero to 53 points. We formed two groups of teachers: High CLP PD -- all 06-07 teachers with 22 or more points Low CLP PD -- a sample of those teachers with 3 or fewer points matched to the high CLP PD group as closely as possible on grade level and student demographics (see page 2 of report)

13. Evaluation Design Our goal was to then examine the achievement of students in each of these two teacher groups.

14. Think/Pair/Share Think about a comprehensive professional development program with which you are familiar. Would this type of design be useful in evaluating that program? Why or why not?

15. Student Demographics Despite our best efforts to match the student groups... Compared to students in the Low CLP PD group, a higher percentage of the students in the High CLP PD group were: On free/reduced meal status ELL Hispanic Therefore, on average students in the High CLP PD group would be predicted to perform less well than those in the Low CLP PD group.

16. Teacher Characteristics How did the two groups of teachers compare on characteristics other than the amount of CLP professional development? Compared to the High CLP PD teachers, the Low CLP PD teachers on average: Were more experienced Had more overall graduate credits / clock hours

17. Findings— Reading On K-6 measures of reading performance, students in the High CLP PD group performed higher than students in the Low CLP PD group on nine of the ten measures available (see page 3 of report).

18. Findings— Math If the teachers in the High CLP PD group were just “better teachers,” then we would see similar results in math. On measures of math performance in grades 2- 6, students in the High CLP PD group performed on average higher than students in the Low CLP PD group on three of the six measures available (see page 3 of report).

19. Findings— Writing On the Grade 4 Writing WASL, students in the High CLP PD group performed slightly lower than students in the Low CLP PD group.

20. Findings— Science The Science WASL requires a great deal of reading to be successful. On the Grade 5 Science WASL, a much higher percentage of students in the High CLP PD group met standard than students in the Low CLP PD group.

21. Possible Sources of Error Comparing demographics grade-by-grade showed that the High CLP PD group did not have a higher percentage of low income students at every grade. Scoring of the district assessments might not be applied consistently across all classrooms.

22. Combining WASL Data Combining WASL data avoids some issues: Overall student demographics for grades 3-6: -- 41% of High CLP PD students were low income. -- 38% of Low CLP PD students were low income. WASL scoring is consistent across all classrooms.

23. Combining WASL Data Combining Reading WASL Grades 3-6: 73% of High CLP PD students met standard 66% of Low CLP PD students met standard Combining Math WASL Grades 3-6: 58% of High CLP PD students met standard 57% of Low CLP PD students met standard

24. Summary of Study 1 High CLP PD students performed better on practically every reading measure compared to Low CLP PD students. This was true despite the fact more High CLP PD students were from low income homes, and High CLP PD teachers were less experienced than the other group. The High CLP PD students did not perform better in math, showing that these students and teachers were not simply “more competent” than the Low CLP PD group.

25. Study 2 Student Achievement in the 07-08 School Year

26. Goal of Study 2 To try to replicate the results of Study 1, using the same model.

27. The CLP PD Continuum Our teachers in Study 2 had CLP PD points that ranged from zero to 78 points. We formed two new groups of teachers: High CLP PD -- with 26 or more points Low CLP PD -- a sample of those teachers with 6 or fewer points matched to the high CLP PD group as closely as possible on grade level and student demographics (see page 5 of report)

28. Student Demographics Again, the High CLP group had more students from traditionally low achieving groups (low income, Hispanic, ELL). The two groups at each of the grade levels were closer in demographics than in Study 1.

29. Teacher Demographics The two groups of teachers were closer in experience and graduate credits/clock hours than those in Study 1. A slight edge was in the direction of the High CLP teachers in Study 2. Some teachers were in both studies:  25 teachers in the High CLP group  13 teachers in the Low CLP group

30. Study 2 Findings Findings for Study 2 were more inconsistent than in Study 1 (see page 6 of report). Again, students in the High CLP group showed more strength in reading and science than in math or writing compared to the Low CLP group.

31. Summary of Findings A summary of the findings from both studies is on page 7 of the report.

32. Evaluating the Model “High CLP” teachers were identified only by the number of activities in which they participated, not by actual observation of their classroom practices. The inference is that they use effective literacy practices as a result of the CLP PD activities. The activities tallied as CLP PD points included only those provided through district coaching staff. Some schools provided CLP training through their school staff.

33. Evaluating the Model Were the “High CLP” teachers better at teaching reading before participating in CLP activities? Data from the 2005 WASL (“pre” CLP) indicates that students of teachers who later became “High CLP” actually did less well in reading than those of teachers who later became “Low CLP.”

34. Next Steps Narrow the PD activities Conduct classroom observations Focus on intermediate grades Focus on schools with high ELL Use expertise of “High CLP” teachers to help us refine the model.

35. Group Sharing What designs have you used to evaluate comprehensive professional development programs? What have you found to be the strengths and weaknesses of designs you have used?