1. Expert Panel Discussion: Measuring Fidelity as a Method for Evaluating Technical Assistance Technical Assistance & Dissemination Program Meeting Jill Lammert & Jill Feldman, Westat April 28, 2015 1

2. Meet the Panelists!  Jill Lammert & Jill Feldman, Westat (Moderators)  Dean Fixsen, State Implementation and Scaling Up of Evidence Based Practices (SISEP)  Lynn Holdheide, CEEDAR Center  Rob Horner, Positive Behavioral Interventions & Supports (PBIS)  Amy McCart, Schoolwide Integrated Framework for Transformation (SWIFT)  Kristin Reedy, National Center on Systemic Improvement (NCSI)

3. Goals for the Discussion  Expose participants to a range of approaches for measuring fidelity of TA.  Pique interest among participants in creating fidelity frameworks for their TA projects.  Provide opportunities for participants to ask experts for guidance and advice. 3

4. Definition of Terms: Fidelity of Implementation vs. Fidelity of Intervention  Fidelity of implementation answers: –Are key components of a program being implemented as intended?  Fidelity of intervention answers: –Did the degree of adoption of key components lead to desired outcomes? 4

5. Sample Research Questions  Fidelity of Implementation –Were all elements of the technical assistance delivered by program staff? –Were all of the mentoring services provided to related services professionals according to the specified TA plan?  Fidelity of Intervention –To what extent did related services professionals engage in the mentoring services with the TA team? –To what extent did the mentoring meetings cover the specified TA topics? Were professionals satisfied with the quantity and quality of the mentoring services provided? 5

6. Measuring Fidelity in the Context of Evaluating TA 6 Measuring Fidelity Recipients’ Use of New Knowledge & Skills Recipients’ Learning Recipients’ Reactions Organizational Support & Change Ultimate Outcomes

7. Framework for Creating a Fidelity Measurement System 1.Specify the program logic model 2.Operationally define key components 3.Select data sources and measures 4.Establish fidelity thresholds 5.Calculate fidelity ratings 6.Determine how ratings will be used 7

8. Sample TA Logic Model 8

9. Questions for Panelists  In what ways does your expert judgment guide how you think about measuring fidelity of TA? –What to measure? –What data to use? –How to approach calculating ratings? –How to decide which ratings to consider as evidence of faithful implementation?  How do you approach measuring fidelity of TA components/activities that are intended to vary or be adapted (e.g., coaching at a range of intensity levels)? 9

10. Questions for Audience  Based on what you’ve heard today, what do you think will be the most challenging for your project in devising a fidelity framework?  What issues raised by the panelists seem particularly relevant for your project?  Based on what you heard today, how would you approach creating a logic model and identifying the key components for your TA project? 10

11. 11 Questions?

12. Contact Information Jill Lammert JillLammert@westat.com Jill Feldman JillFeldman@westat.comJillFeldman@westat.com ww.cippsite.org 12