1. Building Implementation Capacity to Improve Youth Outcomes Allison Metz, Ph.D. Associate Director National Implementation Research Network University of North Carolina at Chapel Hill

2. Agenda The Challenge Implementation Science Plan for Change Active Implementation Frameworks

3. The Challenge: Recognizing the Gaps Science to Service Gap – What is known to be effective is not what is selected to help students Implementation Gap – What is selected is not used with fidelity and good outcomes – What is used with fidelity is not sustained for a useful period of time – What is used with fidelity is not used on a scale sufficient to broadly impact youth outcomes

4. Implementation Gap RESEARCH PRACTICE Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions. IMPLEMENTATION

5. RESEARCH PRACTICE GAP Why Focus on Implementation? IMPLEMENTATION “Children, youth, and families cannot benefit from interventions they do not experience.”

6. EffectiveNOT Effective Effective NOT Effective IMPLEMENTATION INTERVENTION Actual Benefits (Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999) Inconsistent; Not Sustainable; Poor outcomes Unpredictable or poor outcomes; Poor outcomes; Sometimes harmful from Mark Lipsey’s 2009 Meta-analytic overview of the primary factors that characterize effective juvenile offender interventions – “... in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”

7. “Implementation science is the systematic study of variables and conditions that lead to full and effective use of evidence-based programs and other effective innovations in typical human service settings.” —Blase and Fixsen, 2010 National Implementation Research Network “Implementation Science” Download at: http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

8. Best data show these methods, when used alone Do not Result in Implementation as Intended – Diffusion/ Dissemination of information – Training – Passing laws/ mandates/ regulations – Providing funding/ incentives – Organization change/ reorganization 5 to 10% return on investment NECESSARY BUT NOT SUFFICIENT

9. Plan for Change: “Making It Happen” Letting it happen – Recipients are accountable Helping it happen – Recipients are accountable Making it happen – Purposeful use of implementation practice and science – Implementation teams are accountable —Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

10. To successfully implement and sustain evidence-based and evidence-informed youth violence prevention interventions, we need to know: WHAT to do What is the intervention (e.g.identified prevention, intervention and enforcement strategies that are feasible and relevant)? HOW to do it Active and effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults) WHO will do it Organized, purposeful, & active implementation support from linked implementation teams

11. Q. How? A. Effective Implementation Changing the behavior of practitioners and administrators Creating the setting conditions to facilitate these changes Creating the processes to maintain and improve these changes in both setting conditions and behavior of well- intentioned adults So that youth benefit

12. Active Implementation Frameworks Implementation Drivers Implementation Stages Improvement Cycles Implementation Teams

13. IMPLEMENTATION DRIVERS Common features of successful supports to help make full and effective use of a wide variety of innovations

14. Core Implementation Components © Fixsen & Blase, 2008 Positive Outcomes for Youth Competency Drivers Organization Drivers Leadership Effective Youth Violence Prevention Practices Capacity to provide direction and vision Staff capacity to support youth with the selected practices Institutional capacity to support practitioners in implementing practices with fidelity

15. Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Adaptive Technical Integrated & Compensatory Competency Drivers Organization Drivers Leadership Improved youth outcomes Interventions meet Implementation Consistent Use of Prevention Innovations © Fixsen & Blase, 2008

16. IMPLEMENTATION STAGES Purposeful matching of critical implementation activities to the stage of the process

17. Stages AND Drivers Implementation Takes Time: 2 – 4 Years EXPLORATION INSTALLATION INITIAL IMPLEMENTATION FULL IMPLEMENTATION Drivers “DRIVERS”

18. Stages of Implementation EXPLORATION Competency Drivers Organization Drivers Leadership Drivers Integrated & Compensatory —Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 “Pay now or Pay later”

19. Goals of Exploration Create readiness for change Changing hearts and minds Examine degree to which the proposed strategies and practices meet the needs of our community and our youth Determine whether the strategies, practices, and implementation are desirable and feasible

20. The Hexagon An EBP Exploration Tool NEED FIT RESOURCES EVIDENCE CAPACITY READINESS Fit with current Initiatives School, district, state priorities Organizational structures Community values Need in school, district, state Academic & socially significant Issues Parent & community perceptions of need Data indicating need Resources and supports for: Curricula & Classroom Technology supports (IT dept.) Staffing Training Data Systems Coaching & Supervision Administration & system Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe Several replications How well is it operationalized? Are Imp Drivers operationalized? The “Hexagon” can be used as a planning tool to evaluate evidence- based programs and practices during the Exploration Stage of Implementation. Download available at: www.scalingup.org/tools-and-resources EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4. HighMedLow Need Fit Resource Availability Evidence Readiness for Replication Capacity to Implement Total Score © National Implementation Research Network 2009-2012 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland

21. Stages of Implementation Competency Drivers Organization Drivers Leadership Drivers Integrated & Compensatory INSTALLATION —Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 “If you build it, they will come”... but you actually have to build it!

22. Goals of Installation Structural and functional changes are made to support implementation Staff selection protocols developed First ‘practitioners’ selected Define and initiate training of first cohort of practitioners Develop coaching system and plans Evaluate readiness and sustainability of data systems (e.g. fidelity, outcomes)

23. Stages of Implementation Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 EXPLORATION INSTALLATION INITIAL IMPLEMENTATION “Get Started, then Get Better.” Competency Drivers Organization Drivers Leadership Drivers Integrated & Compensatory

24. Goals of Initial Implementation Work through the Awkwardness Provide training and coaching on the evidence-based practice, re-organization of school roles, functions and structures Make use of improvement cycles to resolve systems issues Continue buy-in efforts and manage expectations All the components of the program or innovation are at least partially in place and the implementation supports begin to function

25. Stages of Implementation 2 - 4 Years Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 FULL IMPLEMENTATION “The only thing worse than failing and not knowing why you failed, is succeeding and not knowing why you succeeded.”

26. Goals of Full Implementation Maintaining and improving skills and activities throughout the system Components integrated, fully functioning Skillful practices by front line staff, supervisors, administrators (50% meet performance criteria) Changes in policy are reflected in practice at all levels Ready to be evaluated for expected outcomes

27. IMPROVEMENT CYCLES Changing on purpose to support the new way of work

28. PDSA Cycles Improvement Cycles Plan DoStudy Act

29. New practices do not fare well in existing organizational structures and systems Effective innovations are changed to fit the system, as opposed to existing systems changing to support effective innovations. People, organizations, and systems... Cannot change everything at once (too big; too complex; too many of them and too few of us) Cannot stop and re-tool (have to create the new in the midst of the existing) Cannot know what to do at every step (we will know it when we get there) Many outcomes are not predictable (who knew!?)

30. Plan-Do-Study-Act Cycles  Rapid cycle problem solving (Shewhart; Deming)  Transformation Zone  Usability testing (Neilson; Rubin)  Practice-policy communication loops

31. Practice-Policy Communication Cycle Policy Practice Policy Enables Practices Plan Do External Implementation Support Policy Practice Structure Procedure Practice Informs Policy Feedback Study - Act FORM SUPPORTS FUNCTION

32. IMPLEMENTATION TEAMS Organized, expert assistance to develop and sustain an accountable and effective structure

33. Implementation Teams Provide accountable and effective structure to move intervention through stages of implementation Scope of the initiative determines the number of teams and the linked communication protocols needed Focus is on – Ongoing “buy-in” and readiness – Installing and sustaining the Implementation Drivers – Fidelity & Outcomes – Systems Alignment and Stage-based work – Problem-solving and sustainability

34. “We tend to focus on snapshots of isolated parts of the system and wonder why our deepest problems never seem to get solved.” —Senge, 1990 Linked Team Structures Implementation Teams Site-based Implementation Team Site-based Implementation Team Community-based Implementation Team Community-based Implementation Team Regionally-based Implementation Team State-based Implementation Team State-based Implementation Team

35. Who Makes Change Happen? Implementation Teams INDICATORS CORE COMPETENCIES Representative Accountable Team Structure Formal/Practice Knowledge Model fluency Fully operationalize Intervention Know the Intervention Best practices for Implementation Core components Stage-appropriate Work Know Implementation Communicate Change Use of data for decision making, problem solving and feedback loops Know Improvement Cycles Knowledge and skills for system building and components Know Systems Change SO THAT... …can effect change throughout the system and make decisions … can promote implementation of core components, adaptations, and infrastructure …can guide implementation and build capacity throughout the organization and system …can develop and follow- through on action planning …can support efforts to improve access, reach or scale, improve connection, influence decision- making

36. Examples of Success Implementation results (higher fidelity) and intervention results (improved outcomes) improve when Active Implementation Frameworks are used purposefully and systematically Catawba County Child Wellbeing Project- 5 year period to develop, implement and evaluate post-care service system for children exiting foster care; high fidelity and early indicators of success Colorado EPIC Project- Scale-up of MI using Active Implementation Frameworks in corrections EPIS Center at Penn State University - 419 age-grade cohorts over a 5-year period: youth in CTC communities using EBPs had significantly lower rates of delinquency, greater resistance to negative peer influence, stronger school engagement and better academic achievement Evidence-Based Model Purveyors – NFP, MST, FFT use many components of active implementation frameworks to achieve sustainable outcomes

37. Summary: “Making it Happen” for youth  Purposeful selection of an effective and feasible “What”  Conceptualize a change process so that effective interventions for children and families can become embedded and sustained in socially complex settings  “stage-matched activities” to guide the process  “implementation drivers” to build the infrastructure  Improvement processes are critical  the work is never done because the environment is in motion  Invest in the development of organized, “expert” implementation support

38. Stay Connected! nirn.fpg.unc.edu www.scalingup.org www.implementationconference.org Allison.metz@unc.edu nirn@unc.edu