Robyn Mildon, PhD Executive Director Centre for Evidence and Implementation

Key messages Public policy decisions should be based on evaluations of programs that have been implemented with quality. Otherwise, the relative value and cost-effectiveness of alternative programs cannot be determined. Implementation is important for all programs and services and increasing the quality of implementation increases the chances that these will yield its intended outcomes. Durlak (2013)

Key messages It is possible to adapt an evidence-based program to fit local circumstances and needs as long as the program’s core components, established by theory or preferably through empirical research, are retained and not modified. High quality implementation is the joint responsibility of multiple stakeholders who typically include funders/policy makers, program developers/researchers, local practitioners, and local administrators. Durlak (2013)

People cannot benefit from something they do not receive

Only 9% of those who completed all 3 phases of training followed through to implementation.

Effective interventions and services + Effective implementation = Positive outcomes

Implementation science addresses the “know-do” gap in social services and health care: the disparity between what scientific research identifies as the best evidence-based practices, and what is actually done in the community. (Merrert et al 2016) Studies suggest that it takes an average of 17 years for 14 % of original research to be integrated into physician practice and that only 54 % of US adults receive care that meets indicators of high quality (McGlynn et al; 2003)

Implementation science is the study of methods to promote the use and integration of research evidence into policy and practice (Lobb and Codlitz, 2013) Implementation is the use of strategies to adopt and integrate evidence-based health, mental health, and social care interventions and change practice patterns within specific settings

Evidence to practice gap Widespread implementation with sustainment has been difficult to achieve across human services What is implemented often disappears with time and staff turnover. Moving effective interventions from development and research settings to the practice setting, with fidelity and good effect, involves far more then making the interventions available (Fixsen et al, 2005; Kauffman Foudation Best Practices Project; Rogers 1995)

Alone, these will not do. Identification and cataloguing of “evidence-based” practices and programs. Making laws and policy directives Providing funding One way dissemination of information (tip sheets, websites, lectures like this one...) Implementation without changing supporting roles and functions does not work Training alone (Embry & Biglan, 2008; Fixsen et al., 2005 )

Implementation matters

“Evidence” on effectiveness helps you select what to implement for whom “Evidence” on these outcomes does not help you implement the program or practice Fixsen & Blase (2008)

Implementation Matters (from Fixsen et al., 2005) Implementation: The How Effective Not Effective Inconsistent; not sustainable; poor outcomes Effective Actual benefits Intervention – The What Poor outcomes; sometimes harmful Not Effective Poor outcomes (Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

Durlak & DuPre (2008) Review of over 500 implementation studies in the field of prevention and health promotion on programs for children and youth the magnitude of mean effect sizes were at least 2 to 3 4mes higher when programs were implemented well with few or no problems in the implementation full implementation of programs was associated with better outcomes, particularly when fidelity and dosage were used to measure levels of implementation.

”… in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented. …” Lipsey (2009)

Letting it happen – Diffusion; networking; communication Helping it happen – Dissemination; manuals; websites Making it happen (Implementation Science) – Purposeful and proactive use of implementation practice and science Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

Implementation frameworks

Quality Implementation Framework

Phase One : Initial Considerations regarding the host setting (elements) 1. Conducting a Needs and Resources Assessment 2. Conducting a Fit Assessment 3. Conducting a Capacity/ Readiness Assessment 4. Possibility for Adaptation 5. Obtaining Explicit Buy-in from Critical Stakeholders and Fostering a Supportive Organizational Climate 6. Building General/Organisational Capacity 7. Staff recruitment/ maintenance 8. Effective Pre-innovation Staff Training  

Phase Two : Creating a Structure for Implementation 9 Phase Two : Creating a Structure for Implementation 9. Creating Implementation Teams: 10. Developing an implementation plan: Phase three : Ongoing Structure Once Implementation Begins 11. Technical assistance/ Coaching/Supervision: 12. Process evaluation 13. Supportive feedback mechanism Phase Four : improving Futures Applications 14. Learning from experience

Implementation frameworks

Staged and phased approach to implementation Operational implementation Sustained Implementation Exploration Installation Assess needs Examine fit and feasibility Operationalise model Make decisions Stakeholder buy-in Shared vision Implementation supports developed Practitioners trained in new processes Make necessary structural and instrumental changes Identify coordinated yet differentiated roles Service delivery initiated Data used to drive decision-making and continuous improvement Rapid cycle problem solving Feedback loops among stakeholders Shared authority and decision- making Skillful implementation System and organisational changes institutionalised Child and family outcomes measureable Shared resources Shared data Culture and values shift PRC to lead This diagram shows the implementation stages that are based on the ‘implementation science’ methodology successfully employed by PRC in similar projects. It emphasises the critical important of the exploration and installation phases for ensuring a rigorous and sustainable implementation. Implementation process for complex change is around two years with appropriate support

Implementation strategies

Implementation Strategies “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice” Proctor, Powell & McMillen (2013)

Improved outcomes for children and families Effective implementation (fidelity) Coaching Systems Intervention Organization Drivers Competency Drivers Training Integrated & Compensatory Leadership So let’s try to put this into context. We start with what we’re trying to achieve – our outcome. To do that, we need to use what we think will work – the evidence-based practices that make up the IFSS Practice Model. To make sure we’re using the Practice Model well, and that the workers are supported to do that job, we put infrastructure in place to make that happen as easily as possible for the workers. What we have here is the core components or building blocks of the infrastructure needed to support practice, organsiational and systems change. They are drawn from research around the successfully implemented programs and practices. There are three categories of Implementation Drivers: Competency, Organization, and Leadership When these core components are in place they provide the support to establish and maintain a successful implementation of an evidence-based practice or other innovation. Competency Drivers are mechanisms that help to develop, improve, and sustain practitioners’ and supervisors’ ability to implement an intervention to benefit the client. Competency Drivers include: Staff selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective child welfare services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Technical Adaptive 28

Accountability Questions Relevant Lit 1 Needs/Resources What are the underlying needs & conditions that must be addressed? Needs/Resource Assessment 2 Goals What are the goals, target population, & objectives? Goal Setting 3 Best Practice What evidence based models & best practice programs can be used in reaching the goals? Consult Literature on Science Based & Best Practice Programs 4 Fit What actions need to be taken so the selected program “fits” the community context? Feedback on Comprehensiveness & Fit of Program 5 Capacities What organizational capacities are needed to implement the program? Assessment of Organizational Capacities 6 Plan What is the plan for this program? Planning 7 Process Is the program being implemented with quality? Process evaluation 8 Outcome Evaluation How well is the program working? Outcome and Impact Evaluation 9 Improve How will continuous quality improvement strategies be included? Continuous Quality Improvement 10 Sustain If the program is successful, how will it be sustained? Sustainability and Institutionalization

TOWARD AN EVIDENCE INFORMED SYSTEM FOR INNOVATION SUPPORT Training + QI/QA + Tools + TA + = Current Level of Capacity + GTO Steps: (1) Needs & Resources; (2) Goals & Desired Outcomes; (3) Science-based practices; (4) Fit; (5) Capacity ; (6) Plan; (7) Implementation & Process Evaluation; (8) Outcome evaluation; (9) Continuous Quality Improvement; and (10) Sustainability Actual Outcomes Achieved To Achieve Desired Outcomes From Barbee et al 2011.

Thank you For more information please contact Associate Professor Robyn Mildon Executive Director +61 488 188 072 Robyn.mildon@cei.org.au www.cei.org.au