Measuring Implementation and Intervention Fidelity in Scaling Up: Processes, Tools, and Benefits Carol M. Trivette, Ph.D. Karen Blase, Ph.D. Frank Porter Graham Child Development Institute UNC at Chapel Hill. North Carolina OSEP Project Directors Conference July 2012 Center for Early Literacy Learning Center on Everyday Child Language Learning Orelena Hawks Puckett Institute Asheville and Morganton, North Carolina

Through the CELL and SISEP Lens…. GOALS: Describe differences between intervention and implementation processes Describe the differences between intervention fidelity and implementation fidelity Illustrate how a “cascading logic model” details implementation strategies and outcomes at different levels Connect implementation fidelity to improved intervention fidelity.

Intervention Practices and Fidelity Intervention practices include methods and strategies used by intervention agents (teachers, therapists, clinicians, parents, etc.) to affect changes or produce desired outcomes in a targeted population or group of recipients (e.g., children with disabilities). Therefore, intervention fidelity refers to the degree to which evidence-based intervention practices are used as intended by early childhood practitioners, teachers, parents, or other intervention agents and have expected or intended. Dunst, C. J., Trivette, C. M., McInerney, M., Holland-Coviello, R., Masiello, T., Helsel, F., & Robyak, A. (2008). Measuring training and practice fidelity in capacity-building scaling-up initiatives. CELLpapers, 3(1), Available at

Implementation Practices and Fidelity Implementation practices include methods and procedures used by implementation agents (trainers, coaches, instructors, supervisors, etc.) to promote interventionists’ use of evidence-based intervention practices. Therefore implementation fidelity refers to the degree to which coaching, in-service training, instruction, or any other kind of evidence-based professional development practice is implemented as intended and has the effect of promoting the adoption and use of evidence-based intervention practices. Trivette, C. M., & Dunst, C. J. (2011, August). Implementation with fidelity: How to get changes in early childhood classroom practices. Paper presented at the Global Implementation Conference, Washington, DC.

© Fixsen & Blase, 2008 All Students & Families School Teachers and Staff State Department Leadership State Department Leadership Building Implementation Team Implementation- Skilled Workforce Develop staff with special implementation skills Re-Purpose “District” Implementation Teams Regional Implementation Teams Adult interactions produce Student benefits

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes SISEP Cascading TA Logic Model

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes SISEP Cascading TA Logic Model

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes Teachers and StaffProvision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions Teachers and staff competently and confidently use evidence-based practices ~ Intervention Fidelity~ SISEP Cascading TA Logic Model Intervention Fidelity: The degree to which evidence-based intervention practices are used as intended by early childhood practitioners, teachers, parents, or other intervention agents and have expected or intended. Dunst, C. J., Trivette, C. M., McInerney, M., Holland-Coviello, R., Masiello, T., Helsel, F., & Robyak, A. (2008).

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions Teachers and staff competently and confidently use evidence-based practices ~ Intervention Fidelity~ Building and District staffProvision of services from the Regional Implementation Team to LEA (District and School staff) Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff ~Implementation Fidelity~ SISEP Cascading TA Logic Model Implementation Fidelity: The degree to which coaching, in-service training, instruction, or any other kind of evidence-based professional development practice is implemented as intended and has the effect of promoting the adoption and use of evidence-based intervention practices. Trivette, C. M., & Dunst, C. J. (2011, August). Implementation with fidelity: How to get changes in early childhood classroom practices. Paper presented at the Global Implementation Conference, Washington, DC.

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions Teachers and staff competently and confidently use evidence-based practices ~ Intervention Fidelity~ Local and District staffProvision of services from the Regional Implementation Team to LEA (District and School staff) Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff ~ Implementation Fidelity Regional Implementation Team Members Skillful leadership by State Transformation Specialists, supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams. Timely and skillful provision of services by Regional Implementation Teams to LEA SISEP Cascading TA Logic Model

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions Teachers and staff competently and confidently use evidence-based practices ~ Intervention Fidelity~ Building, District, Regional StaffProvision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff Regional Implementation Team Members Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams. Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities State Transformation Specialists State Management Team 1 st Generation Regional Implementation Team Intensive technical assistance by SISEP to develop implementation capacity at multiple levels Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams. SISEP

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions Teachers and staff competently and confidently use evidence-based practices ~ Intervention Fidelity~ Building, District, Regional StaffProvision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff Regional Implementation Team Members Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams. Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities State Transformation Specialists State Management Team 1 st Generation Regional Implementation Team Intensive technical assistance by SISEP to develop implementation capacity at multiple levels Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams. SISEP Implementation Fidelity

PopulationIntervention Strategies (WHAT) Intervention Outcomes All students and/or young children Teachers and staff fully and effectively use evidence-based practices Improved academic, behavioral outcomes PopulationsImplementation Strategies (HOW) Implementation Outcomes Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions Teachers and staff competently and confidently use evidence- based practices ~ Intervention Fidelity~ Building, District, Regional StaffProvision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff Regional Implementation Team Members Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams. Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities State Transformation Specialists State Management Team 1 st Generation Regional Implementation Team Intensive technical assistance by SISEP to develop implementation capacity at multiple levels Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams. SISEP Implementation Drivers Best Practices & DCA Implementation Drivers Best Practices & DCA Regional Implementation Team Capacity Assessment (RIT CA) State Capacity Assessment (SCA & RIT CA) SCA & RIT CA All Measures Above RIT Capacity Assessment

Implementation and Intervention No intervention practice, no matter its evidence base, is likely to be adopted and used if the implementation methods used to teach or train practitioners to use the practice are not themselves effective. Therefore concern for the characteristics of implementation practices that are associated with optimal learner and practitioner outcomes should be of paramount importance as part of implementation research. Dunst, C. J., & Trivette, C. M. (2009). Let's be PALS: An evidence-based approach to professional development. Infants and Young Children, 22(3), doi: /IYC.0b013e3181abe16

Center for Early Literacy Learning Center on Early Childhood Language Learning CELL – OSPE TA&D Project – Children 0-3 with disabilities Developed evidence-based early literacy practices for young children with disabilities (web site) Worked with 6 states to scale up these early literacy practices Worked with state teams to develop training State teams trained trainers or teachers or practitioners Teachers and parents provided intervention with children CECLL – OSEP Model Demonstration – Children 0-3 with disabilities Provided training to Part C EI providers in 3 states Part C provided coaching with parents Parents provided intervention to children

Tiered Approach to Training to Promote Adoption and Sustained Use of CELL One Many State Resource (Leadership) Team Regional Resource (Cohorts) Team Level I Local/Regional Trainers Level II End Users – Teachers or Parents Interventions to Children

Research Synthesis of Adult Learning Studies a Research synthesis of studies of accelerated learning, coaching, guided design, and just-in-time training 58 randomized control design studies 2,095 experimental group participants and 2,213 control or comparison group participants Combination of studies in university and nonuniversity settings Learner outcomes included learner knowledge, practices, skills, attitudes, and self-efficacy beliefs The influence of the adult learning methods on the learner outcomes was estimated by weighted Cohen’s d effect sizes for the differences on the post test scores for the intervention vs. nonintervention group participants a Dunst, C.J., Trivette, C.M., & Hamby, D.W. (2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. International Journal of Continuing Education and Lifelong Learning, 3(1),

Characteristics Used to Code and Evaluate the Implementation Studies a Planning Introduce Engage the learner in a preview of the material, knowledge, or practice that is the focus of instruction or training Illustrate Demonstrate or illustrate the use or applicability of the material, knowledge, or practice for the learner Application Practice Engage the learner in the use of the material, knowledge, or practice Evaluate Engage the learner in a process of evaluating the consequence or outcome of the application of the material, knowledge, or practice Deep Understanding Reflection Engage the learner in self-assessment of his or her acquisition of knowledge and skills as a basis for identifying “next steps” in the learning process MasteryEngage the learner in a process of assessing his or her experience in the context of some conceptual or practical model or framework, or some external set of performance standards or criteria a Donovan, M. et al. (Eds.) (1999). How people learn. Washington, DC: National Academy Press.

Most Effective Adult Learning Method Practices CharacteristicPractice Mean Effect Size IntroductionOut-of-class learner activities/self-instruction 0.64 Classroom/workshop presentations 0.63 Pre-class learner exercises 0.54 IllustrationTrainer role playing/simulations 0.55 Learner informed input 0.53 PracticingReal-life learner application 0.94 Real-life learner application/role playing 0.86 EvaluationSelf assessment of strengths/weaknesses 0.94 ReflectionIdentify performance-improvement goals 1.27 Journaling/behavior suggestions 0.82 MasteryStandards-based assessment 0.86

Cumulative Effects of Different Combinations of the Most Effective Adult Learning Method Practices NUMBER OF ADULT LEARNING METHOD PRACTICES

PALS (Participatory Adult Learning Strategy) PLAN APPLICATION RECYCLE Active Learner Involvement Reflection and Mastery Practice and Evaluate Introduce and Illustrate Identify Next Steps in the Learning Process INFORMED UNDERSTANDING

CELL Trainer Feedback Form Content Items 1.The importance of literacy-rich environments was well explained by the trainer. 2.The key characteristics of responsive teaching were described and illustrated in ways that made the instructional practice easy to understand. Training-Method Items 3.The importance of active participation of trainees as a method was made explicitly clear. 4.The training method was described in enough detail to understand its key elements. Practice Items 5.A sufficient amount of time was devoted to each component of the CELL literacy model. 6.The activities for engaging trainees in learning the CELL literacy practices were especially helpful.

Hypothesized Relationships Among the Fidelity Measures Variations in implementation fidelity should be related to variations in intervention fidelity. Tests of the hypothesis include evaluation of the relative importance of the quantity and quality of implementation fidelity, and the interactions between the types and elements of fidelity. Variations in intervention fidelity should be related to variations in practice outcomes. Tests of the hypothesis include evaluation of the relative importance of the quantity and quality of intervention fidelity, and the interactions between the types and elements of fidelity.

26 Relationship Between Implementation Fidelity and Child Communication Development Purpose Evaluate the relationship between variations in the fidelity of parents’ use of the CECLL practices and changes in the children’s early communication development Participants Sixty-three infants and toddlers 8 to 40 months of age and their parents Outcome Measures Fidelity: A weekly parent-completed scale of child participation in and the characteristics of everyday learning activities Child Outcome: Early Communication Development Scale

27 CECLL Staff Training of Practitioners Practitioner Implementation with Parent Parent Use of the Intervention With the Child Enhanced Language Learning Implementation Fidelity Intervention Fidelity Child Outcomes Hypothesized Relationships Between the Implementation and Intervention Fidelity Measures and Child Outcomes

28 Parent-Completed Fidelity Measure The scale used to assess intervention fidelity captured two aspects of the use of the CECLL practices: Quantity of child participation in everyday activities (number of settings, number of activities within settings, frequency of participation) Quality of child participation in everyday activities (development-enhancing characteristics of the child learning experiences and opportunities)

29 Influences of Fidelity on Child Communication Development Cohen’s d = 0.62 for differences in the slopes

Summary Intervention Fidelity Matters to Achieve Outcomes Implementation Fidelity Matters to Achieve Intervention Fidelity Multiple levels of the system need to be impacted to achieve these outcomes Cascading logic models and measures at every level can help develop effective implementation strategies and intervention outcomes.