1. CaMSP Qualitative Findings and Best Practices Results from the Evaluation Year 3 Report (2006-07) Presented at the CDE CaMSP Network Meeting November 3, 2008 Sacramento Patty O’Driscoll Scott Phelps

2. Who Are We? Public Works, Inc. a nonprofit in Pasadena dedicated to working with communities, government, schools and parents by providing services and resources to educate and inform children, youth and families. Our work is in three areas: Education Reform Workforce Development Intervention/Prevention

3. Research Questions 1.How have the Partnerships ensured that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses? 2.How have the Partnerships enhanced the quality of the mathematics and science teacher workforce? 3.What evidence-based outcomes from the Partnerships contribute to our understanding of how students effectively learn mathematics and science?

4. Statewide Evaluation Approach Process measures focus on how a program is implemented and the outcome measures focus on the results of the program or intervention. Public Works, Inc. uses both quantitative and qualitative data collection methods Purpose: find trends across the Partnerships for a statewide picture and best practices discovered throughout the state

5. Statewide Evaluation Approach Website to collect Teacher Data and log Partnership Activities Partner Survey Participating Teacher Survey Site Visits (15-17 per year) Teacher Outcomes through Web database (teacher characteristics, program attendance) Student Outcomes (control/treatment) Final Report each year…..

6. Levels of Evaluation Teacher/Student Assessment (Participants) District/School Evaluation (Teachers, Principals, Administrators) Local Evaluation of the Partnership (Evaluator and Leadership Team) State Evaluation (CDE/Public Works, Inc.) Federal Evaluation (USDOE)

7. Components of Qualitative Evaluation Partner Survey (involvement, roles and responsibilities) Teacher Survey (opinions and impact on teaching practice)

8. Components of Qualitative Evaluation Site Visits –2 Days –Subset of sites (15-20) usually new cohorts in first year –Interviews of multiple partners related to 5 key features –PD Observation Telephone interviews –New this year

9. Key Features, Findings & Best Practices Features of the programs reviewed: Partnership-driven Teacher quality Challenging courses and curricula Evidence-based design and outcomes Institutional change and sustainability

10. Partnership-Driven Feature Represents its target population of districts, teachers and institutions of higher education Commitment and engagement to the partnership Governance and equality of partners in planning and logistics Target population of the grant is served

11. Partnership-Driven Findings Partnerships are becoming more streamlined Concrete and committed partners--high quality PD Partners involved in previous cohorts helpful to recruiting and retaining teachers and can hit the ground running Strong coordination and strong district personnel

12. Partnership-Driven Best Practices Whittier and CSU Dominguez Hills PROMISE embeds coaching and continuous improvement

13. Teacher Quality Feature Cohort of teachers that are enrolled in all aspects of the professional development Target number and hours are being met Approach to professional development is tied to state standards and is focused on both improving the content knowledge and pedagogical approach of teachers Research-based, high quality PD for both intensive training and classroom follow-up components

14. Teacher Quality Findings Recruitment and retention of cohort is challenging but improving Differentiation of Intensive and Classroom Follow-up more clear High satisfaction with training More explicit integration with standards and textbooks needed Very little emphasis on teacher college credit

15. Teacher Quality Best Practices San Benito Rigorous Coaching Model RAAFA Lesson Study North San Mateo County Coaching Vallejo Observation and Coaching Content Lesson Study in Alameda San Diego City Intensive Lesson Study and K-12 Alliance

16. Challenging Curricula and Courses Feature Professional development is aligned to standards and aimed at transforming and improving instruction The project is creating new challenging courses, lessons and curricula for pre- service or existing teachers and/or students New courses, curricula, expectations or experiences for Students will result from the professional development teachers receive in this grant

17. Challenging Courses & Curricula Findings Hoping that program will result in new courses for students and teachers Some connections being made between districts and IHE training Exposure to collaborative curriculum development at most partnerships A few partnerships were offering challenging online online teacher courses or supplemental courses or assignments

18. Challenging Courses & Curricula Best Practices San Benito PROJECT Smart use of local resources CSSP in Orange County integration of EL strategies and coaching ExCEL in Fullerton Institute and Summer Lab School San Diego COE Online PD Summer School Pacing Guides

19. Institutional Change & Sustainability Feature The role of the IHE is clear and integral to the project. The institutions (IHE and Districts) is impacted by the project in terms of a tangible result. The IHE is involved in the planning, curriculum development and delivery. Education department and the discipline department (math/science) are involved. Teachers can receive credit for their professional development. Sustainability

20. Institutional Change & Sustainability Findings Involvement of both IHE disciplines strengthens the partnership Teachers appreciate access to IHE content specialties CaMSP raising priority of mathematics and science Professors improved their practices Want to sustain coaching and lesson study but still working on how…

21. Institutional Change & Sustainability Best Practices TAPS/Sustainability Teacher Apprenticeship Project placing undergraduate students Pre-Services Courses IHE Professor and Department Change Chula Vista and USD Master’s Level Programs

22. Evidence Based Design & Outcomes Feature There is a research or evidence-based model for their professional development Evaluation plan makes sense for the project The design and measurement system will produce an impact on teacher and classroom quality. There is a cohort of teachers that make sense to examine the impact on student outcomes A pre-/post- assessment of teacher knowledge is conducted and local student assessment is conducted (above and beyond CSTs)

23. Evidence-Based Design & Outcomes Findings Impact of grant on teachers and students continues to be difficult to measure for local evaluators Teacher feedback is positive but knowing the impact on student outcomes is a challenge Partnerships are responsive to state evaluation requests

24. Evidence-Based Design & Outcomes Student Assessment Across Districts in CSSP Targeting a grade level as groundwork for local evaluation Video Clip Analysis in Lesson Study Local Student Assessment in the multi- district North State Mathematics Partnership

25. Year 3 Report Outline Introduction Findings related to five key features (integrated site visit and survey with more focus on Cohort 3) Best practices Teacher Outcomes: Cohort 3 Student Outcomes from control/treatment group of teachers on statewide measures from Cohort 1 and Cohort 2 Available at www.publicworksinc.org

26. Recommendations 1.Continue to help sites refine structure of model and encourage consistency in “intensive” and “classroom follow-up” activities 2.Focus on the integration of other statewide initiatives in mathematics and science at the state and local level and with other policymakers

27. Recommendations 3.Encourage involvement of IHEs across departments that align to long-term needs of teacher workforce in mathematics and science 4.Provide support to districts (especially small and mid-size) to sustain professional development efforts and build professional development capacity

28. Recommendations 5. Strengthen local research/evaluation projects 6. Provide technical assistance in student and teacher assessment 7. Improve alignment of contract period, reporting and research design for Cohorts 8. Improve statewide data collection and accessibility