1. Robyn Mildon, PhD Executive Director Centre for Evidence and Implementation

2. Key messages
Public policy decisions should be based on evaluations of programs that have been implemented with quality. Otherwise, the relative value and cost-effectiveness of alternative programs cannot be determined.
Implementation is important for all programs and services and increasing the quality of implementation increases the chances that these will yield its intended outcomes.
Durlak (2013)

3. Key messages
It is possible to adapt an evidence-based program to fit local circumstances and needs as long as the program’s core components, established by theory or preferably through empirical research, are retained and not modified.
High quality implementation is the joint responsibility of multiple stakeholders who typically include funders/policy makers, program developers/researchers, local practitioners, and local administrators.
Durlak (2013)

5. People cannot benefit from something they do not receive

6. Only 9% of those who completed all 3 phases of training followed through to implementation.

7. Effective interventions and services + Effective implementation = Positive outcomes

8. Implementation science addresses the “know-do” gap in social services and health care:
the disparity between what scientific research identifies as the best evidence-based practices, and what is actually done in the community. (Merrert et al 2016)
Studies suggest that it takes an average of 17 years for 14 % of original research to be integrated into physician practice and that only 54 % of US adults receive care that meets indicators of high quality (McGlynn et al; 2003)

9. Implementation science is the study of methods to promote the use and integration of research evidence into policy and practice (Lobb and Codlitz, 2013)
Implementation is the use of strategies to adopt and integrate evidence-based health, mental health, and social care interventions and change practice patterns within specific settings

10. Evidence to practice gap
Widespread implementation with sustainment has been difficult to achieve across human services
What is implemented often disappears with time and staff turnover.
Moving effective interventions from development and research settings to the practice setting, with fidelity and good effect, involves far more then making the interventions available
(Fixsen et al, 2005; Kauffman Foudation Best Practices Project; Rogers 1995)

11. Alone, these will not do.
Identification and cataloguing of “evidence-based” practices and programs. Making laws and policy directives Providing funding One way dissemination of information (tip sheets, websites, lectures like this one...) Implementation without changing supporting roles and functions does not work Training alone (Embry & Biglan, 2008; Fixsen et al., 2005 )

12. Implementation matters

13. “Evidence” on effectiveness helps you select what to implement for whom
“Evidence” on these outcomes does not help you implement the program or practice
Fixsen & Blase (2008)

14. Implementation Matters (from Fixsen et al., 2005)
Implementation: The How
Effective
Not Effective
Inconsistent; not sustainable; poor outcomes
Effective
Actual benefits
Intervention – The What
Poor outcomes; sometimes harmful
Not Effective
Poor outcomes
(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

15. Durlak & DuPre (2008)
Review of over 500 implementation studies in the field of prevention and health promotion on programs for children and youth
the magnitude of mean effect sizes were at least 2 to 3 4mes higher when programs were implemented well with few or no problems in the implementation
full implementation of programs was associated with better outcomes, particularly when fidelity and dosage were used to measure levels of implementation.

16. ”… in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented. …”
Lipsey (2009)

17. Letting it happen – Diffusion; networking; communication
Helping it happen – Dissemination; manuals; websites
Making it happen (Implementation Science) – Purposeful and proactive use of implementation practice and science
Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

18. Implementation frameworks

20. Quality Implementation Framework

21. Phase One : Initial Considerations regarding the host setting (elements)
1. Conducting a Needs and Resources Assessment
2. Conducting a Fit Assessment
3. Conducting a Capacity/ Readiness Assessment
4. Possibility for Adaptation
5. Obtaining Explicit Buy-in from Critical Stakeholders and Fostering a Supportive Organizational Climate
6. Building General/Organisational Capacity
7. Staff recruitment/ maintenance
8. Effective Pre-innovation Staff Training

22. Phase Two : Creating a Structure for Implementation 9
Phase Two : Creating a Structure for Implementation 9. Creating Implementation Teams: 10. Developing an implementation plan: Phase three : Ongoing Structure Once Implementation Begins 11. Technical assistance/ Coaching/Supervision: 12. Process evaluation 13. Supportive feedback mechanism Phase Four : improving Futures Applications 14. Learning from experience

23. Implementation frameworks

24. Staged and phased approach to implementation
Operational implementation
Sustained Implementation
Exploration
Installation
Assess needs
Examine fit and feasibility
Operationalise model
Make decisions
Stakeholder buy-in
Shared vision
Implementation supports developed
Practitioners trained in new processes
Make necessary structural and instrumental changes
Identify coordinated yet differentiated roles
Service delivery initiated
Data used to drive decision-making and continuous improvement
Rapid cycle problem solving
Feedback loops among stakeholders
Shared authority and decision- making
Skillful implementation
System and organisational changes institutionalised
Child and family outcomes measureable
Shared resources
Shared data
Culture and values shift
PRC to lead
This diagram shows the implementation stages that are based on the ‘implementation science’ methodology successfully employed by PRC in similar projects.
It emphasises the critical important of the exploration and installation phases for ensuring a rigorous and sustainable implementation.
Implementation process for complex change is around two years with appropriate support

26. Implementation strategies

27. Implementation Strategies “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice” Proctor, Powell & McMillen (2013)

28. Improved outcomes for children and families
Effective implementation (fidelity)
Coaching
Systems Intervention
Organization Drivers
Competency Drivers
Training
Integrated & Compensatory
Leadership
So let’s try to put this into context.
We start with what we’re trying to achieve – our outcome.
To do that, we need to use what we think will work – the evidence-based practices that make up the IFSS Practice Model.
To make sure we’re using the Practice Model well, and that the workers are supported to do that job, we put infrastructure in place to make that happen as easily as possible for the workers.
What we have here is the core components or building blocks of the infrastructure needed to support practice, organsiational and systems change.
They are drawn from research around the successfully implemented programs and practices.
There are three categories of Implementation Drivers: Competency, Organization, and Leadership
When these core components are in place they provide the support to establish and maintain a successful implementation of an evidence-based practice or other innovation.
Competency Drivers are mechanisms that help to develop, improve, and sustain practitioners’ and supervisors’ ability to implement an intervention to benefit the client. Competency Drivers include: Staff selection, Training, Coaching, and Performance Assessment
Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective child welfare services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention
Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive
Technical
Adaptive 28

30. Accountability Questions Relevant Lit1
Needs/Resources
What are the underlying needs & conditions that must be addressed?
Needs/Resource Assessment 2
Goals
What are the goals, target population, & objectives?
Goal Setting 3
Best Practice
What evidence based models & best practice programs can be used in reaching the goals?
Consult Literature on Science Based & Best Practice Programs 4
Fit
What actions need to be taken so the selected program “fits” the community context?
Feedback on Comprehensiveness & Fit of Program 5
Capacities
What organizational capacities are needed to implement the program?
Assessment of Organizational Capacities 6
Plan
What is the plan for this program?
Planning 7
Process
Is the program being implemented with quality?
Process evaluation 8
Outcome Evaluation
How well is the program working?
Outcome and Impact Evaluation 9
Improve
How will continuous quality improvement strategies be included?
Continuous Quality Improvement 10
Sustain
If the program is successful, how will it be sustained?
Sustainability and Institutionalization

31. TOWARD AN EVIDENCE INFORMED SYSTEM FOR INNOVATION SUPPORT
Training +
QI/QA +
Tools +
TA + =
Current
Level of
Capacity +
GTO Steps: (1) Needs & Resources; (2) Goals & Desired Outcomes; (3) Science-based practices; (4) Fit; (5) Capacity ; (6) Plan; (7) Implementation & Process Evaluation; (8) Outcome evaluation; (9) Continuous Quality Improvement; and (10) Sustainability
Actual Outcomes Achieved To
Achieve
Desired
Outcomes
From Barbee et al 2011.

35. Thank you For more information please contact
Associate Professor Robyn Mildon
Executive Director