1. Evidence-Informed Planning: Benefiting from Evidence- Based Interventions Defending Childhood January 26, 2011 Melissa K. Van Dyke, LCSW Associate Director National Implementation Research Network University of North Carolina, Chapel Hill

2. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Two Sides of the Same Coin To successfully implement and sustain evidence-based and evidence-informed interventions, we need to know: The WHAT - What is the intervention (e.g. Al’s Pals, FFT, PCIT, Second Step) AND The HOW - Effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults)

3. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 The Challenge “It is one thing to say with the prophet Amos, ‘Let justice roll down like mighty waters’ … William Sloane Coffin Social activist and clergyman … and quite another to work out the irrigation system.”

4. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Science “to” Service SCIENCE SERVICE GAP Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions. Why Focus on Implementation? “Children and families cannot benefit from interventions they do not experience.” IMPLEMENTATION

5. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Goals for Today’s Session The “What” Review general information about evidence- based practices The “How” Present ‘stage-related’ work necessary for successful service and system change Present the Implementation Drivers that result in competence and sustainability Explore “improvement cycles” and how to use them at a number of levels The “Who” Discuss the roles and responsibilities of implementation team and program purveyors

6. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Which intervention is right for you? What are the needs of your population? What interventions are available to address those needs? What is the strength of the evidence of those interventions? Which interventions are a good fit for our community? Do we have what is required to fully and effectively implement these interventions?

7. EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4. HighMediumLow Need Fit Resources Availability Evidence Readiness for Replication Capacity to Implement Total Score: Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland

8. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Becoming an Informed Consumer NREPP http://www.nrepp.samhsa.gov/ - Descriptive information - Outcomes - Quality of Research - Study Population - Readiness for Dissemination - Costs - Replications Questions to ask model developers or model purveyors: http://www.nrepp.samhsa.gov/pdfs/Questions _To_Ask_Developers.pdf

9. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 NREPP Program Review Sample For example – Second Step Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale) Implementation Materials4.0 Training and Support4.0 Quality Assurance3.5 Overall Rating 3.8

10. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Science-to-Service Gap Implementation Gap What is adopted is not used with fidelity and good outcomes What is used with fidelity is not sustained for a useful period of time What is used with fidelity is not used on a scale sufficient to impact social problems

11. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Implementation Review and synthesis of the implementation research and evaluation literature (1970 – 2004) Multi-disciplinary Multi-sector Multi-national

12. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Insufficient Methods Implementation by laws/ compliance by itself does not work Implementation by “following the money” by itself does not work Implementation without changing supporting roles and functions does not work Diffusion/dissemination of information by itself does not lead to successful implementation Training alone, no matter how well done, does not lead to successful implementation Fixsen, Naoom, Blase, Friedman, Wallace, 2005

13. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Implementation Pre-Requisites Start with Data related to Need Look for “best evidence” to Address the Need An Evidence-Based Practice or Program An Evidence-Informed Initiative Systems Change and Its Elements Clearly operationalize the program and/or practice features or the systems change elements Operationalize Part of Speech: verb Definition: to define a concept or variable so that it can be measured or expressed quantitatively Webster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7) Copyright © 2003-2008 Lexico Publishing Group, LLC

14. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 What Works EffectiveNOT Effective Effective NOT Effective IMPLEMENTATION INTERVENTION Actual Benefits (Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999) Inconsistent; Not Sustainable; Poor outcomes Poor outcomes; Sometimes harmful

15. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 What Works EffectiveNOT Effective Effective NOT Effective IMPLEMENTATION INTERVENTION Actual Benefits (Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999) Inconsistent; Not Sustainable; Poor outcomes Poor outcomes; Sometimes harmful from Mark Lipsey’s 2009 Meta- analytic overview of the primary factors that characterize effective juvenile offender interventions – “... in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”

16. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Implementation Frameworks Practice, program and systems change through… Multi-dimensional, fully integrated use of Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles

17. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Implementation Drivers Common features of successful supports to help make full and effective uses of a wide variety of innovations Staff Competency Organizational Supports Leadership

18. © Fixsen & Blase, 2008 Performance Assessment (fidelity measurement) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Integrated & Compensatory Competency Drivers Organization Drivers Leadership Improved outcomes for children and families Implementation Drivers

19. © Fixsen & Blase, 2008 Coaching Training Selection Competency Drivers Graphics by Steve Goodman,2009 Implementation Lens Implementation Drivers Performance Assessment (fidelity measurement) Improved outcomes for children and families

20. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Performance Assessment Purposes: Measure fidelity Ensure implementation Reinforce staff and build on strengths Feedback to agency on functioning of Recruitment and Selection Practices Training Programs (pre and in-service) Supervision and Coaching Systems Interpretation of Outcome Data

21. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Coaching Purposes: Ensures fidelity Ensures implementation Develops clinical and practice judgment Provides feedback to selection and training processes Grounded in “Best Practices”

22. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Training and Coaching OUTCOMES % of Participants who Demonstrate Knowledge, Demonstrate New Skills in a Training Setting, and Use new Skills in the Classroom TRAINING COMPONENTS Knowledge Skill Demonstration Use in the Classroom Theory and Discussion 10%5%0%..+Demonstration in Training 30%20%0% …+ Practice & Feedback in Training 60% 5% …+ Coaching in Classroom 95% Joyce and Showers, 2002

23. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Training Purposes: “Buy-in” Knowledge acquisition Skill Development

24. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Selection Purposes: Select for the “unteachables” Screen for pre-requisites Set expectations Allow for mutual selection Improve likelihood of retention after “investment” Improve likelihood that training, coaching, and supervision will result in implementation

25. © Fixsen & Blase, 2008 Coaching Training Selection Competency Drivers Graphics by Steve Goodman,2009 Implementation Lens Implementation Drivers Performance Assessment (fidelity measurement) Improved outcomes for children and families

26. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Organizational Change "All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling Business Expert

27. © Fixsen & Blase, 2008 Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Competency Drivers Organization Drivers Graphics by Steve Goodman,2009 Implementation Drivers Performance Assessment (fidelity measurement) Improved outcomes for children and families

28. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Decision Support Data System Purposes: To make a difference for children and families Provide information to assess effectiveness of evidence-based practices Analyze the relationship of fidelity to outcomes To guide further program development Engage in continuous quality improvement Interaction with Core Implementation Components Celebrate success Be accountable to consumers and funders

29. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Facilitative Administration Purposes: Facilitates installation and implementation of the Drivers Aligns policies and procedures Takes the lead on Systems Interventions Looks for ways to make work of practitioners and supervisors easier!!

30. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Systems Intervention Purposes: Identify barriers and facilitators for the new way of work Create an externally and internally “hospitable” environment for the new way of work Contribute to cumulative learning in multi-site projects.

31. © Fixsen & Blase, 2008 Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Adaptive Technical Integrated & Compensatory Competency Drivers Organization Drivers Leadership Graphics by Steve Goodman,2009 Implementation Drivers Performance Assessment (fidelity measurement) Improved outcomes for children and families

32. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Integrated and Compensatory Implementation Drivers Integrated Consistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI) Compensatory At the practitioner level At the program level

33. Performance Assessment Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Adaptive Technical Integrated & Compensatory Competency Drivers Organization Drivers Leadership Improved outcomes for children and families Major Implementation Initiatives occur in stages: Exploration (Sustainability) Installation (Sustainability) Initial Implementation (Sustainability) Full Implementation (Sustainability & Effectiveness) Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 Implementation Takes Time 2 - 4 Years

34. Stages of Implementation Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 EXPLORATION Integrated & Compensatory Competency Drivers Organization Drivers Leadership

35. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration Goals: Examine degree to which the Evidence Based Practice, best practice, systems change meets the needs in the settings identified Determine whether moving ahead with the initiative and implementation is desirable and feasible Create readiness for change at many levels “Pay now or pay later.”

36. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Sustainability Goals: Financial: Ensure funding streams for desired change and necessary infrastructure Programmatic: Ensure high fidelity and positive outcomes through infrastructure improvement and maintenance Plan for turnover “The only thing harder than getting there is staying there.”

37. Stages of Implementation Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 EXPLORATION INSTALLATION Integrated & Compensatory Competency Drivers Organization Drivers Leadership

38. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Installation Goal: To make the structural and instrumental changes necessary to initiate services “If you build it, they will come”... but you actually have to built it!

39. Stages of Implementation Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 EXPLORATION INSTALLATION INITIAL IMPLEMENTATION Integrated & Compensatory Competency Drivers Organization Drivers Leadership

40. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Initial Implementation Goals: Survive the awkward stage! Learn from mistakes Continue “buy-in” efforts Manage expectations “Anything worth doing…is worth doing poorly.”

41. Stages of Implementation 2 - 4 Years Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 EXPLORATION INSTALLATION INITIAL IMPLEMENTATION FULL IMPLEMENTATION Integrated & Compensatory Competency Drivers Organization Drivers Leadership

42. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Full Implementation Goals: Maintaining and improving skills and activities throughout the system Components integrated, fully functioning Skillful practices by front line staff, supervisors, administrators Changes in policy that are reflected in practice at all levels Ready to be evaluated for expected outcomes “The only thing worse than failing and not knowing why you failed, is succeeding and not knowing why you succeeded.” ~ Jane Timmons-Mitchell

43. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Stages of Implementation Major Implementation Initiatives occur in stages: Exploration (Sustainability) Installation (Sustainability) Initial Implementation (Sustainability) Full Implementation (Sustainability & Effectiveness) Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

44. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration “Many implementation efforts fail because someone underestimated the scope or importance of preparation. Indeed, the organizational hills are full of managers who believe that an innovation’s technical superiority and strategic importance will guarantee acceptance.” Leonard-Barton & Kraus, Harvard Business Review, 1985

45. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Analyze data related to “needs”  Identify options and assess feasibility  Reassess, revise, prioritize, re- scope  Formalize structures CREATE READINESS

46. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Analyze data related to “needs”  Identify options and assess feasibility  Reassess, revise, prioritize, re- scope  Formalize structures CREATE READINESS

47. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Form an “Exploration Workgroup” Formation of an exploration workgroup Focal point for the exploration work Empowered to make decisions and/or to make recommendations Representative of the “stakeholders” Develop collaboration / co- ownership in the community

48. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Analyze data related to “needs”  Identify options and assess feasibility  Reassess, revise, prioritize, re- scope  Formalize structures CREATE READINESS

49. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Analyze Data Related to “Needs” Assessment of current outcomes Dimensions – Root cause analysis (5 Whys) Prevalence of the problem(s) – How frequent and pervasive? Persistent nature of the problem – Have we been struggling for a long time? Social significance – If this changed, would it make a significant different for students? Leverage point – If these few indicators changed then other outcomes would be likely to be “pulled along.”

50. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Analyze data related to “needs”  Identify options and assess feasibility  Reassess, revise, prioritize re- scope  Formalize structures CREATE READINESS

51. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Identify Options &Assess Feasibility Needs Fit Resource availability Evidence Readiness for replication or degree to which it is operationalized Capacity

52. Initiative : 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4. HighMediumLow Need Fit Resources Availability Evidence Readiness for Replication Capacity to Implement Total Score: Need in State, District, Schools Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State, District, School Priorities Organizational structures Community Values Resource Availability Resources Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Time Evidence – is there any? Outcomes – Is it worth it? Fidelity or process data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Fit and Feasibility of Initiatives Readiness Qualified purveyor Expert TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency and Departments Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland

53. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Analyze data related to “needs”  Identify options and assess feasibility  Reassess, revise, prioritize, re- scope  Formalize structures CREATE READINESS

54. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Does this change initiative still address the most critical needs? Does it fit our current political and social context? Do we have the necessary resources and support? Do we have the capacity and access to necessary expertise to proceed? Have we bitten off more than we can chew? Is this our leverage point? What has emerged during Exploration that impacts our decisions? Reassess...

55. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Analyze data related to “needs”  Identify options and assess feasibility  Reassess, revise, prioritize, re- scope  Formalize structures CREATE READINESS

56. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Formalize Structures Formalize structures and processes – develop implementation teams based on… Design elements (e.g. components of the initiative) Legal Practitioner level Agency level District/State/Tribal work Stakeholders Identify linkages among the ‘structures’

57. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Exploration: In Depth What happens during Exploration?  Form “exploration workgroup”  Review data related to “needs”  Identify options and feasibility  Reassess, revise, re-scope  Formalize structures CREATE READINESS

58. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Resistance to Change There is no such thing – only inadequate preparation It is not “their” problem, it is ours.

59. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Creating Readiness for Change Individual readiness for change Transtheoretical Model or Stages of Change Precontemplation Contemplation Preparation Action Maintenance Prochaska and DiClemente

60. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Stages of Change Stage of Change for Pre-Action Individuals: Precontemplation – 40% Contemplation – 40% Preparation – 20% “If only 20% of employees in organizations are prepared to take action....” Janice M. Prochaska, James O. Prochaska, and Deborah A. Levesque (2001)

61. Stages of Implementation 2 - 4 Years Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 EXPLORATION INSTALLATION INITIAL IMPLEMENTATION FULL IMPLEMENTATION Integrated & Compensatory Competency Drivers Organization Drivers Leadership

62. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 “Who” Purveyors Intermediary Organizations Technical Assistance Centers Implementation Teams

63. Organized, Implementation Support Provider Agency Supports Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) Practitioner Competence State and Tribal Leadership Regional Authority Supports Developers Technical Assistance Implementation Team Simultaneous, Multi-Level Interventions

64. Organized, Implementation Support Provider Agency Supports Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) Practitioner Competence State and Tribal Leadership Regional Authority Supports Developers Technical Assistance Implementation Team Simultaneous, Multi-Level Interventions “We tend to focus on snapshots of isolated parts of the system and wonder why our deepest problems never seem to get solved. (Senge, 1990)

65. Systems trump programs! …Patrick McCarthy, Annie E. Casey " All organizations are designed, intentionally or unwittingly, to achieve precisely the results they get.” …R. Spencer Darling System Change

66. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Changing on Purpose New practices do not fare well in existing organizational structures and systems Effective practices are changed to fit the system, as opposed to existing systems changing to support effective evidence- based practices.

67. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 EXISTING SYSTEM Effective approaches are Changed to Fit the System Or Operate in the Shadows (Ghost System) Effective System Change EXISTING SYSTEM IS CHANGED TO SUPPORT THE EFFECTIVENESS OF THE APPROACH (Host System) EFFECTIVE APROACH

68. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Changing on Purpose People, organizations, and systems... Cannot change everything at once (too big; too complex; too many of them and too few of us) Cannot stop and re-tool (have to create the new in the midst of the existing) Cannot know what to do at every step (we will know it when we get there) Many outcomes are not predictable (who knew!?) Trial & Learning

69. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990) Plan – Decide what to do Do – Do it (be sure) Study – Look at the results Act – Make adjustments Cycle – Do over and over again until the intended benefits are realized PDSA Cycles: Trial & Learning

70. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Improvement Cycle Uses Rapid Cycle Teams Problem-solving Practice Improvement Usability Testing Practice-Policy Feedback Loops Transformation Zones

71. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 Implications Clearly understand/define the “What” Stage-matched activities guide the process Build processes/systems to continuously improve “drivers” Local and/or state systems will need time to implement effectively Support the development of organized, skilled implementation support to build organization and system capacity to implement well

72. Copyright © Dean L. Fixsen and Karen A. Blase, 2010 For More Information Melissa Van Dyke melissa.vandyke@unc.edu http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/