1. Measuring Fidelity in Early Childhood Scaling-Up Initiatives: A Framework and Examples Carl J. Dunst, Ph.D. Orelena Hawks Puckett Institute Asheville, North Carolina Presentation made at the “Fidelity: How Much Devotion is Enough? (and Other Methodological Virtues)” session at the Office of Special Education Program’s Project Directors Conference, Washington, DC, July 2008

2. Purposes of the Presentation Describe a framework for measuring fidelity as part of scaling-up early literacy learning practices. Define different types of fidelity at different levels of training and practice. Illustrate how variations in fidelity can be related to variations in practice adoption and outcomes.

3. Center for Early Literacy Learning The main aims of CELL are to: (1) Synthesize research evidence on effective early literacy practices, (2) develop evidence-based practices from the research syntheses, (3) evaluate the validity and use of the practices, and (4) conduct both generalized and specialized technical assistance promoting adoption and use of CELL practices.

4. From Evidence-Based Practices to Fidelity to Outcomes Developing Evidence-Based Early Literacy Learning Practices Scaling-Up the Use of the Practices Measuring the Fidelity of Use of the Practices Child Outcomes

5. Early Literacy Learning

6. Specialized Technical Assistance

7. Framework for Measuring Fidelity

8. Types of Fidelity Implementation Diffusion Intervention

9. Implementation Fidelity Implementation fidelity is defined as the degree to which CELL training activities of known characteristics are implemented as planned and promote participant understanding of the known characteristics of the evidence- based practices constituting the focus of training.

10. Diffusion Fidelity Diffusion fidelity is defined as the degree to which second and third generation technical assistance providers use CELL training methods and procedures of known characteristics to promote end-user understanding of the key characteristics of evidence-based practices.

11. Intervention Fidelity Intervention fidelity is defined as the extent to which end users adopt and use instructional practices of known characteristics for implementing evidence-based practices of known characteristics.

12. Elements of Fidelity Each type of fidelity includes two elements: Fidelity of the training processes (how) Fidelity of practice understanding and adoption (what)

13. Elements of Fidelity Training processes are HOW we promote adoption and use of WHAT we want implemented by end-users. As state by Fixsen et al. (2005), it is important to be aware of the difference between the practices constituting the focus of training and the training methods used to promote adoption of the practices.

14. Framework for Structuring Fidelity Assessments

15. CELL Fidelity Checklist for Measuring Implementation and Diffusion Fidelity Fidelity of both implementation and diffusion training is being measured by a CELL fidelity checklist that assesses the degree to which CELL training processes and CELL practices are accomplished with (1) sufficient dose (2) and with ample quality exposure to (3) key training and practice characteristics, and (4) are viewed by participants as socially valid.

17. Measuring Fidelity of Intervention Fidelity of intervention is measured in terms of practitioner or parent, or both, use of instructional practices and evidence-based practices of known characteristics as part of the provision of everyday early childhood intervention. Fidelity is measured by the frequency and number of practices that are used and the development-enhancing qualities (active ingredients, core features, etc.) of the intervention practices. The social validity of both the practices and the child outcomes from using the practices are also assessed.

21. Relationship Between Fidelity and Child Outcomes Practice Measures Low Users High Users Mean F-TestCohen’s a a d Number of Weeks -.21.16 1.26.37 Number of Learning Games -.32.32 5.70*.67 a Mean standardized score. *p <.05. Game Characteristics -.61.51 5.73* 1.32

22. Further Evaluation of the Effectiveness of the Practice Additional information about the effectiveness of the practices was determined by a post-test only quasi- experimental between group design that included two outcomes, one of which was a “dependant variable that a treatment should affect” and a second dependant measure that was not expected to be related to the use of the practice (Shadish et al., 2002, p.184). Obtaining an expected group by outcome measure interaction was used to ascertain effectiveness.

24. Conclusions Fidelity is important as part of ascertaining the adoption and use of evidence-based practices. A fidelity framework can help structure the development and collection of fidelity measures. It is important to measure both implementation and intervention fidelity. Relating variation in fidelity to variation in outcomes can help identify what matters most in terms of targeted practices.

25. For More Information Visit www.puckett.org for methods and procedures for identifying evidence-based practices www.earlyliteracylearning.org for information on the Center for Early Literacy Learning.