1. A Multi-Level Framework to Understand Factors Influencing Program Implementation in Schools Celene E. Domitrovich, Ph.D. Penn State Prevention Research Center Implementation Methods Meeting September 20, 2010

2. Co-Authors & Funding Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buck;ey, J. A., Olin, S., Romanelli, L. H., Leaf, P. J., Greenberg, M. T., & Ialongo, N. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6-28. NIMH and NIDA (P30 MH06624) CDC (1U49CE 000728 and K01 CE001333-01) NIDA (R01 DA019984)

3. Current Challenges Broad dissemination and “going to scale” with evidence-based intervention Maintaining effectiveness under “real-world conditions Understanding the factors that promote or undermine implementation quality throughout the dissemination process

4. General Definition of Implementation: What an intervention consists of in practice and the degree to which it was conducted as it was originally intended (Durlak, 1995; Yeaton & Sechrest, 1981)

5. Defining the Model: Intervention Core Elements Standardization Delivery Support System Core Elements Standardization Delivery

6. Planned vs. Actual Intervention

7. Planned vs. Actual Support System

8. Measuring the Quality of the Intervention and Support System Measuring the Quality of the Intervention and Support System Adherence/Fidelity Dosage Quality of Delivery

9. Factors Outside the Model:

10. Factors Outside the Implementation Model Individual Level Professional & Psychological Characteristics Intervention Perceptions & Attitudes School Level Resources, Administrative Leadership and Support Classroom Climate, School Characteristics Macro Level Policies & Financing Leadership & Human Capital University/Community Partnerships

12. Critical Issue #1: Non-linear process There are stages to the diffusion of innovations Implementation quality is critical throughout but very few studies have measured it over time The factors that support or undermine the process may be more or less relevant at each stage

13. Critical Issue #2: Factors Interact Factors associated with implementation quality cannot be examined in isolation They interact both within and across levels In order to understand how these factors influence one another, we need multiple measures assessed simultaneously and over time. Large sample sizes are need to address this and many other implementation research challenges

14. Critical Issue #3: Measurement There are different potential sources of implementation data They vary in how cost effective they are to collect We know very little about who can assess what, when it should be measured, which measures are the most important, and how often implementation quality should be assessed.

15. Critical Issue #4: Lack of randomization Most studies of implementation examine associations between factors and implementation or between implementation and target outcomes within the intervention group which limits conclusions that can be drawn. Interventions often have unique components or processes that are difficult to assess in the control group. Measures that can be used in both conditions provide a way to preserve the RCT design Very few studies actually test theories of implementation by manipulating implementation itself or factors outside the process in randomized trials

16. Critical Issue #5: Multi-component or Integrated Interventions The field is moving towards the use of integrated and adaptive interventions that present unique implementation challenges. How do we capture quality when multiple implementers are involved in the delivery of an intervention? How do we understand implementation in the context of interventions where different participants receive different components?

17. Future Directions Theory-driven research –Implementation interventions Understanding adaptation Improved Measurement Statistical strategies to address challenges