1. Preparing Facilitators to Use and Adapt Video-Based Professional Development Materials with Fidelity Nanette Seago WestEd, Redwood City, CA Karen Koellner Hunter College, New York Jennifer Jacobs University of Colorado, Boulder

2. 2 Presentation Overview I.Fidelity Definitions II.Facilitating Professional Development III.Learning and Teaching Geometry PD (LTG PD) IV.LTG PD Efficacy Study I.Preparing Knowledgeable Facilitators II.Measures and Results V.Suggestions for the field

3. 3 Defining Fidelity DefinitionAuthors The determination of how well an intervention is implemented in comparison with the original program design during an efficacy and/or effectiveness study National Research Council, 2004; U.S. Department of Education, 2006 The extent to which the critical components of an intended program are present when that program is enacted Century, Rudnick & Freeman, 2010 Variation and fidelity are important dimensions of implementation to designers Penuel & Means, 2003 The degree to which a facilitator adheres to the goals and adapts to the participants ‘ and/or context as needed, while maintaining the integrity of the PD design Borko, 2004; Seago, Jacobs & Koellner, 2016

4. 4 Facilitating Professional Development Four elements of a professional development system (Borko, 2004)

5. Learning and Teaching Geometry PD NSF Funded ( #0732757) Built around authentic video clips – 3-6 minutes in length – Grades 5-9 classrooms Video clips intended as objects of inquiry, not exemplars Focus on geometric similarity through the use of video case modules, in which specific and increasingly complex mathematical concepts are presented within the dynamics of classroom practice. Highly-specified PD materials: – Specify in advance particular learning goals – Make explicit design characteristics – Provide extensive supports and resources for facilitators (both mathematical and pedagogical)

6. 6 LTG Efficacy Study New NSF-funded project (#1503399). First year of study. Explores whether the LTG PD program produces a beneficial impact on teachers’ mathematical knowledge, classroom teaching practices, and their students’ knowledge in the domain of geometry. Research question: Can the facilitator implement the LTG PD materials with fidelity? Randomized experimental design, 140 mathematics teachers (grades 6- 12) Intervention: Summer 2016—Week-long institute (30 hours of PD); 24 hours of academic year PD (2016-2017) Trained Knowledgeable Facilitator 2015-2016 (Hannah)

7. 7 Preparing Knowledgeable Facilitators In order to facilitate the LTG PD with fidelity, the facilitator participated in two phases of preparation over 8 months: Phase I: Individually studied the PD materials, session by session, and then discussed each session with the co-PIs/ PD developers Phase II: Rehearsal facilitation observed by co-PIs in order to document fidelity and adaptations.

8. 8 Fidelity Measures The co-PIs analyzed fidelity of the rehearsal by using adapted versions of two existing instruments: LTG PD Session Logging Tool (Horizon Research, Inc,) Measures adherence to suggested time and the extent to which each activity was modified Learning Goals Instrument (Century, 2008; WestEd, 2013) Measures fidelity to the LTG PD program’s stated mathematics goals and mathematical knowledge for teaching goals within each session

9. 9 Results: Session Logging Tool

10. 10 Learning Goals Ratings Scoring Guide

11. 11 Analysis &Results: Learning Goals Data Collection & Analysis: – Three observers rated each session individually – Observers compared ratings and discussed at the end of each day – Observers reconciled any discrepancies in ratings & came to consensus agreement Results: – There was high fidelity to all learning goals

12. Summary of Fidelity Findings Timing: Adjustments were made to approximately half the activity categories Modifications to activities: Very few activities were modified – One example: Hannah highlighted the connection between geometry & algebra because that was of particular interest to her participants Fidelity to learning goals: Perfect scores

13. 13 Suggestions for the Field: Preparing Facilitators Use a two phase process to prepare facilitators Ensure facilitators have time for in- depth study AND rehearsal Open question: can this type of preparation be scaled?

14. 14 Suggestions for the Field: Measuring Fidelity PD facilitators must take context into account PD programs, including well-specified models, are likely to need some modification Measures of PD implementation fidelity should look at the NATURE of the modifications (not just the amount)

15. Thank you! nseago@wested.org kkoellne@hunter.cuny.edu