1. Making the most of what we have: The role of evidence, the importance of context and working together to achieve high quality implementation to improve outcomes for children and families. Robyn Mildon, PhD Director of Knowledge Exchange and Implementation

2. 2 Cradle to Kinder Evaluation Proposal

3. Parenting Research Centre - non-government, non-profit, independent research, program development, implementation specialist and evaluation organisation -support the use of evidence in practice and participatory program development and evaluation - use evidence from the science and practice of implementation to support effective and full implementation of quality services for children and families

4. Where are we at.. The ultimate goal of services and supports in child welfare service systems is to decrease the prevalence of child abuse and neglect and increase child permanency, safety and wellbeing. Over the two decades, thinking about how to accomplish this has involved a number of strategies..

5. 5

6. Evidence-informed programs consist of collections of practices that are performed within known parameters (philosophy, values, service delivery structure, and treatment components) and are put together in the form of a structured manual containing details of procedures and tools (Fixsen et al., 2005). Evidence-informed practices are effective skills, techniques, and strategies that can be used by an individual practitioner. Such practices describe core intervention components that have been shown to reliably produce desirable effects and can be used individually or in combination to form more complex procedures or programs (Embry, 2004).

7. Evidence to practice gap Despite these efforts, widespread implementation with sustainment has been difficult to achieve across human services What is implemented often disappears with time and staff turnover Moving effective interventions from development and research settings to the practice setting, with fidelity and good effect, involves far more then making the interventions available (Fixsen et al, 2005; Kauffman Foundation Best Practices Project; Rogers 1995; )

8. Only 9% of those who completed all 3 phases of training followed through to implementation.

9. Alone, these will not do. Identification and cataloguing of “evidence-based” practices and programs. Making laws and policy directives Providing funding One way dissemination of information (tip sheets, websites, lectures like this one...) Implementation without changing supporting roles and functions does not work Training alone (Embry & Biglan, 2008; Fixsen et al., 2005 )

10. Children and families cannot benefit from something they do not receive 10

11. The get the most out of what we have.. The role of evidence (the what) Evidence-informed interventions and/or practices High quality implementation (the how): Effective implementation frameworks (e.g. strategies to change and maintain behaviour of practitioners and create hospitable organisational systems) The importance of context, within which the programs and practices will be delivered (community, organisations, political context)

12. Effective interventions and practices + Effective implementation = Positive outcomes

13. Necessary conditions Motivation: intent to change - perceived advantages of innovation + culture + climate + resources + leadership + structure Organisational capacity - context, culture, current infrastructure, organisational processes (sufficient staffing, leadership, well established network) Innovation-specific capacity - Knowledge, skills, abilities needed for the innovation; program champion; specific implementation climate support Scaccia, Cook, Lamont, Wandersman, Castellow, Katz & Beidas, in press

14. Implementation...the planned and intentional use of strategies to put into practice effective innovations within real-world settings (Fixsen et al. 2005; Mitchell, 2011). Implementation is a process, not an event, and should be distinguished from adoption, which is defined as the formal decision to use an effective practice/s (Mitchell, 2011).

15. Implementation Science Is the study of methods to promote the integration of research findings and evidence into health, education and human services 15

16. What we choose to do matters The What..

17. Human services field is moving toward providing services that are demonstrably effective as one way to reduce the prevalence of most common problems and increase wellbeing. In the main, efforts to do this have focused on identifying and disseminating evidence-informed programs. What next..

18. Increasing the adoption and implementation of effective programs and practice Start with outcomes Pay attention to the “implementability” of EIPs Considering programs AND a common elements approach Considering core elements and adaptable characteristics

19. Starting with outcomes Outcome: describes how the target population (who) might change based upon successful completion of your strategies. Improved parent and child safety Improved parental and family functioning Promote a secure, predictable and dependable relationship between the parents/children Increase child and parent self-efficacy Promote responsive and warm parenting Improved parenting coping skills Improve parent and child problem-solving skills Increase positive communication Improved child and adult social connections/resources

20. Identify and assess best fit evidence-informed programs/practices 1.Clarify needs of children and families the agency support. 2.Clarify desired aims and outcomes of the program or service. 3.Collect and summarise the evidence base. 4.Map practice. 5.Select and adapt knowledge to local context. 6.Select, tailor and/or design program or Practice Model.

21. 21

22. 22

23. Programs and practices Where we are looking to achieve change outcomes with a particular target group, and a manualised EIP has been shown to achieve this, we should strive to effectively implement that EIP with high fidelity. This will often mean changing the service setting in which the program will be implemented (staff skill, supervision, organisational systems and structures, service frameworks)

24. Challenges with EIPs for service system challenges with respect to implementation in agencies serving children and families with complex and multiple needs. EIPs can be too constraining of practitioners’ practice as a result of rigid manualization and structure (Lyon et al, under review); there are difficulties in implementing an EIPs widely with fidelity and maintaining effectiveness (Embry & Biglan, 2008); in the real-world there are often difficulties in accessing adequate training in a variety of ESIs that have been developed (Kazdin & Blase, 2011) many existing interventions that are treated as EIPs may also have limited effectiveness, modest effect sizes, issues with scalability, limited generalisation, and difficulties with maintaining or sustaining the program without adaptation in child welfare service settings (Embry & Biglan, 2008).

25. A common elements approach Common elements refer to the individual treatment practices that reflect specific content (e.g., psycho-education, exposure, and rewards) that comprise an intervention Treatment elements are selected to match particular child and family characteristics, allowing for practitioners to establish therapeutic alliances, utilize clinical judgment, and still follow evidence-based practice protocols. Practice elements are defined by their content, not by duration, periodicity or location within a manual. (Chorpita, Daleiden, & Weisz, 2005a; Chorpita, Daleiden, & Weisz, 2005b)

26. Common elements - recognizes the benefits to be gained by understanding more about current practice when trying to build effective services across contexts - offers a genuine possibility in terms of facilitating more effective adoption and implementation - conductive to a high level of adaptability to context and reciprocal adaptation between intervention and context, and these may serve to decrease the substantial barriers to adopting EIP’s that exist in provider attitudes, and characteristics of client populations

27. Fidelity and Adaptation One of the most critical challenges in implementation of EIPs - tension between adaptation (i.e., flexibility) and attaining fidelity Adaptation. “To make fit (as for a specific or new use or situation) often by modification” (Aarons et al 2012) 27

28. Fidelity and Adaptation Exposure : amount of an intervention that is offered to the participants in relation to the amount prescribed in the validated intervention model (the number of sessions or hours of programmed activity offered). Adherence: extent to which the techniques/practices were delivered according to the program developer’s specifications for content. Quality of delivery: pertains to practitioners/manager performance on dimensions that are thought to enhance delivery of the intervention (e.g., enthusiasm, style, ability to facilitate client participation, etc.).

32. Implementation matters The How

33. “Evidence” on effectiveness helps you select what to implement for whom “Evidence” on these outcomes does not help you implement the program or practice Fixsen & Blase (2008)

34. Implementation Matters (from Fixsen et al., 2005) Effective Not Effective Effective Not Effective Implementation: The How Intervention – The What (Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999) Inconsistent; not sustainable; poor outcomes Poor outcomes Poor outcomes; sometimes harmful Actual benefits

35. Implementation matters Durlak & DuPre (2008) review of over 500 implementation studies in the field of prevention and health promotion programs for children and youth found that the magnitude of mean effect sizes were at least 2 to 3 times higher when programs were implemented well with few or no problems in the implementation. full implementation of programs was associated with better outcomes, particularly when fidelity and dosage were used to measure levels of implementation.

36. Implementation matters “... in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.” Lipsey et al (2010)

37. descriptions of the implementation process which include key attributes, facilitators, and challenges related to implementation provide an overview of practices that guide the implementation process and provide guidance to researchers and practitioners by describing specific steps to include in the planning and execution of implementation efforts Frameworks for implementation

38. Letting it happen – Diffusion; networking; communication Helping it happen – Dissemination; manuals; websites Making it happen (Implementation Science) – Purposeful and proactive use of implementation practice and science Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010) 38

39. Quality Implementation Framework (Meyers, Durlak and Wandersman, in press) National Implementation Research Network (Fixsen, Naoom, Blase, Friedman & Wallace, 2005) Getting to Outcomes Framework (Wandersman, Imm, Chinman, & Kaffarian, 2000)

40. Quality Implementation Framework

41. Phase One : Initial Considerations regarding the host setting (elements ) 1. Conducting a Needs and Resources Assessment 2. Conducting a Fit Assessment 3. Conducting a Capacity/ Readiness Assessment 4. Possibility for Adaptation 5. Obtaining Explicit Buy-in from Critical Stakeholders and Fostering a Supportive Organizational Climate 6. Building General/Organisational Capacity 7. Staff recruitment/ maintenance 8. Effective Pre-innovation Staff Training

42. Phase Two : Creating a Structure for Implementation 9. Creating Implementation Teams: 10. Developing an implementation plan: Phase three : Ongoing Structure Once Implementation Begins 11. Technical assistance/ Coaching/Supervision: 12. Process evaluation 13. Supportive feedback mechanism Phase Four : Improving Futures Applications 14. Learning from experience

43. Knowledge to Implementation Cycle (based on Fixsen et al 2005)

44. © Fixsen & Blase, 2008 Performance Assessment Coaching Training Staff selection Systems Intervention Facilitative Administration Decision Support Data System Technical Integrated & Compensatory Competency Drivers Organization Drivers Leadership Improved outcomes for children and families Adaptive Effective implementation (fidelity)

45. Implementation drivers Competency Drivers develop, improve, and sustain ability to implement an intervention with fidelity and benefits to consumers. Drivers include: Selection, Training, Coaching, and Performance Assessment Organizational Drivers create and sustain hospitable organizational and systems environments for effective services. Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers methods to manage technical problems

46. TOWARD AN EVIDENCE INFORMED SYSTEM FOR INNOVATION SUPPORT From Barbee et al 2011. To Achieve Desired Outcomes Training + QI/QA + Tools + TA + = Current Level of Capacity + GTO Steps: (1) Needs & Resources; (2) Goals & Desired Outcomes; (3) Science-based practices; (4) Fit; (5) Capacity ; (6) Plan; (7) Implementation & Process Evaluation; (8) Outcome evaluation; (9) Continuous Quality Improvement; and (10) Sustainability Actual Outcomes Achieved

47. Context Matters

48. Implementation is affected by organisational context Relationship between organisational support for EBP, attitudes towards EBP and use of EBPs in practice. Findings: gap between public and private sector organisations regarding innovation and implementation. private agencies provided greater support for EBP implementation staff working for private agencies reported more positive attitudes toward adopting EBPs. organisational support was significantly positively associated with attitudes toward EBP and EBP use in practice. Aarons et al. (2009)

50. Stages of Implementation 50

51. 51

52. 52 Population of concernIntervention Strategies (WHAT) Intervention Outcomes Children and families where neglect is present. IFSS workers skilfully implement the evidence- based practices within the IFSS Practice Model Reduction in child neglect according to the ‘Child Neglect Index’ IFSS Logic Model

53. The logic model continued Population of concern Intervention Strategies (WHAT)Intervention Outcomes Children and families where neglect is present. IFSS workers skilfully implement the IFSS Practice Model Reduction in child neglect according to the ‘Child Neglect Index’ PopulationsImplementation Strategies (HOW)Implementation Outcomes IFSS workersTeam Leaders carefully select staff and arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments IFSS workers competently use the IFSS Practice Model to improve child neglect and strengthen families 53

54. The logic model continued Population of concern Intervention Strategies (WHAT)Intervention Outcomes Families where child neglect is present. IFSS workers skilfully implement the IFSS Practice Model Reduction in child neglect according to the ‘Child Neglect Index’ PopulationsImplementation Strategies (HOW)Implementation Outcomes IFSS workersTeam Leaders carefully select staff and arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments IFSS workers competently use the IFSS Practice Model to improve child neglect and strengthen families Supervisors/Team Leaders Agency managers engage with technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; installation of data systems to monitor fidelity; review and revise agency policies and procedures Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for workers implementing the IFSS Practice Model 54

55. The logic model continued Population of concern Intervention Strategies (WHAT)Intervention Outcomes Families where child neglect is present. IFSS workers skilfully implement the IFSS Practice Model Reduction in child neglect according to the ‘Child Neglect Index’ PopulationsImplementation Strategies (HOW)Implementation Outcomes IFSS workersTeam Leaders carefully select staff and arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments IFSS workers competently use the IFSS Practice Model to improve child neglect and strengthen families Supervisors / Team Leaders Agency managers engage with technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; installation of data systems to monitor fidelity; review and revise agency policies and procedures Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for workers implementing the IFSS Practice Model Provider Agency Managers FaHCSIA funds and supports an Implementation Support Service for IFSS: formal mechanisms and structures created to enhance policy- practice feedback loops (CIT): streamline reporting requirements Agency managers engage with technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; installation of data systems to monitor fidelity; review and revise agency policies and procedures 55

56. Population of concern Intervention, Strategies and Measures (WHAT) Intervention Outcomes Children and families. IFSS workers skilfully implement the IFSS Practice Model Reduction in child neglect according to the ‘Child Neglect Index’ PopulationsPractice, Organisational and Systems Change Strategies (HOW) Implementation Related Outcomes IFSS WorkersTeam Leaders carefully select staff and provide or arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments IFSS workers competently use the IFSS Practice Model to improve child neglect and strengthen families Team LeadersProvider agency managers facilitate agreements with IFSS trainers, quality consultants, and technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; manage the installation of data systems to monitor fidelity and Service outcomes. Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for caseworkers implementing the IFSS Practice Model Provider Agency Managers FaHCSIA develops a common mission for supporting implementation of IFSS: formal mechanisms and structures created to enhance policy-practice feedback loops; streamline reporting requirements; facilitating collaborative relationships to support system infrastructure. Agency leaders ensure access to skilful, timely training, and implementation supports to Team Leaders: creation of hospitable organisations for the IFSS Practice Model 56 PRC IMPLEMENTATION SUPPORT TEAM

57. The logic model continued Population of concern Intervention, Strategies and Measures (WHAT)Intervention Outcomes Children and families. IFSS workers skilfully implement the IFSS Practice Model Reduction in child neglect according to the ‘Child Neglect Index’ PopulationsPractice, Organisational and Systems Change Strategies (HOW) Implementation Related Outcomes IFSS WorkersSupervisors carefully select staff and arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments IFSS workers competently use the IFSS Practice Model to improve child neglect and strengthen families Team LeadersProvider agency managers facilitate agreements with trainers, quality consultants, and technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; manage the installation of data systems to monitor fidelity. Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for caseworkers implementing the IFSS Practice Model Provider Agency Managers PRC Implementation Team develops a specific training program; provides coaches and develops measures and a data capture system so that provider agencies have improved knowledge, skills and resources in the delivery of the IFSS Practice Model Agency leaders ensure access to skilful, timely training, and implementation supports to Team Leaders: and the creation of hospitable organisations for the IFSS Practice Model 57

58. 58

59. Contact Us Level 5, 232 Victoria Parade East Melbourne, Victoria, Australia 3002 E: info@parentingrc.org.au P: +61 3 8660 3500 F: +61 3 8660 3599