Using Implementation Science To Inform Professional Development And Guide Systems Change   Growing Collaborative Systems to Support Children with Disabilities and their Families NETWORKING FORUM Stevens Point Convention Center Wisconsin September 13, 2011 Karen A. Blase, PhD & Dean L. Fixsen, PhD Co-Directors, National Implementation Research Network Senior Scientists at the Frank Porter Graham Child Development Institute, UNC, Chapel Hill, North Carolina (c) Dean Fixsen and Karen Blase, 2004

The Focus on Evidence The “evidence-based movement” is an national experiment to make better use of research findings in typical service settings. The purpose is to produce greater benefits to children, adults, families, and society. Institute of Medicine. (2000). To err is human: Building a safer health system. I. Kohn, J. Corrigan, M. Donaldson (eds.): Washington, DC: National Academies Press. Institute of Medicine - Committee on Quality of Health Care in America. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, D.C.: National Academy Press. National Advisory Mental Health Council Workgroup on Child and Adolescent Mental Health Intervention Development and Deployment. (2001). Blueprint for change: Research on child and adolescent mental health. Washington, DC: National Institute of Mental Health. National Commission on Excellence in Education. (1983). A Nation at Risk: The Imperative for Educational Reform. Washington, DC: U.S. Government Printing Office. National Institute of Mental Health. (1999). Bridging science and service: A report by the National Advisory Mental Health Council's Clinical Treatment and Services Research Workshop. Rockville, MD: National Institute of Mental Health. New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final Report. Rockville, MD: DHHS Pub. No. SMA-03-3832. (c) Dean Fixsen and Karen Blase, 2008 2

Implementation Science “In theory there is no difference between theory and practice; in practice, there is.” variously attributed to Jan La Van De Snepscheut or Albert Einstein or Yogi Berra

There are Challenges Science to Service Gap Implementation Gap What is known is generally not what is adopted Implementation Gap There are not clear pathways to implementation What is adopted often is not used with fidelity and good effect What is implemented disappears over time and with staff turnover (c) Dean Fixsen and Karen Blase, 2008 4

Learning About Effective Implementation Craft knowledge National meeting of EBP Program Developers National meeting of EBP Implementation Sites National meeting of Implementation Researchers Interviews with 64 Program Developers Work with Program Developers, States, Agencies across domains Formal and scientific information Program Efforts and Replication Data Review of evaluation and research literature related to implementation Publication of Implementation Research: A Synthesis of the Literature (2005) WHAT HAVE WE LEARNED? (c) Dean Fixsen and Karen Blase, 2008 5

There are Shared Challenges!! Implementation is implementation is implementation in: Agriculture and Forestry Manufacturing and Business Substance abuse Child welfare, Education, Engineering, Health and Medicine Juvenile Justice, Mental health, Nursing, …and more The commonalities are striking. Fixsen, D., Naoom, S., Blase, K., Friedman, R., & Wallace, F. (2005). Implementation Research: A synthesis of the literature. The National Implementation Research Network, 1 - 125. (c) Dean Fixsen and Karen Blase, 2008 6

Implementation Math Positive Outcomes for Children Effective Interventions The “WHAT” Effective Implementation The “HOW” Positive Outcomes for Children And Families

Implementation Math Positive Outcomes for Children from Mark Lipsey’s 2009 Meta-analytic overview of the primary factors that characterize effective juvenile offender interventions – “. . . in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.” Effective Interventions The “WHAT” Effective Implementation The “HOW” Positive Outcomes for Children And Families Remembering that any number times 0 is 0

Business as Usual ≠ Impact Necessary But Not Sufficient Data Show These Methods, When Used Alone, Do Not Result In Implementation As Intended: Diffusion/ Dissemination of information Training Passing laws/ mandates/ regulations Providing funding/ incentives Organization change/ reorganization Paul Nutt (2002). Why Decisions Fail Necessary But Not Sufficient

What works?…Fidelity Matters Higher Fidelity is correlated with better outcomes across a wide range of programs and practices Education – SWPBIS, HiPlaces, STEP Children’s Services – FFT, MST, Wraparound, TFM Adult Mental Health – ACT, IPS, IDDT Medicine – DOTS, Texas Algorithm, OMAP School-Based MH Prevention Programs - PATHS (c) Dean Fixsen and Karen Blase, 2009

Policy Implications I didn't have potatoes, so I substituted rice. WHAT WORKS ~ IMPLEMENTING PROPERLY MATTERS I didn't have potatoes, so I substituted rice. Didn't have paprika, so I used another spice. I didn't have tomato sauce, so I used tomato paste. A whole can not a half can - I don't believe in waste. My friend gave me the recipe - she said you couldn't beat it. There must be something wrong with her, I couldn't even eat it! ~ Senior Center Newsletter Higher Fidelity is correlated with better outcomes across a wide range of programs and practices Policy Implications IF THERE IS NOT THE “WILL” OR “MEANS” TO DO IT RIGHT YOU WON’T GET RESULTS 2. YOU CAN’T DO A COST-BENEFIT ANALYSIS IF THERE IS NO BENEFIT! (c) Karen Blase, 2005

What works… Successful implementation on a useful scale requires a purveyor An individual or group of individuals representing a program or practice who actively work to implement that practice or program with fidelity and good effect Purveyors accumulate data & experiential knowledge, & become more effective and efficient over time What is one (c) Dean Fixsen and Karen Blase, 2008 12

Implementation Framework Organizational Structures/Culture Practitioner Purveyor Fidelity and Outcome Measures Evidence-based Practices Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 (c) Dean Fixsen and Karen Blase, 2009

Implementation Framework Organizational Structures/Culture Infrastructure (Train, Coach, Evaluate) Purveyor Fidelity and Outcome Measures Evidence-based Practices Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 (c) Dean Fixsen and Karen Blase, 2009

Active Purveyor Role Simultaneous, Multi-Level Interventions Purveyor Agencies/Providers Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) Practitioners Federal Context Local, County, State Context Purveyor Experienced quickly learn…. (c) Dean Fixsen and Karen Blase, 2008 15

Purveyor AND Intermediary Structures and Strategies Simultaneous, Multi-Level Interventions Agency/Provider Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) Practitioners Federal Context Local, County, & State Context Purveyor and Intermediary Emergence of intermediary organizations at state and regional levels (c) Dean Fixsen and Karen Blase, 2009

Implementation Frameworks Multi-dimensional, Fully integrated Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles (c) Dean Fixsen and Karen Blase, 2008 17

Implementation Frameworks Multi-dimensional, Fully integrated Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles (c) Dean Fixsen and Karen Blase, 2008 18

Implementation Drivers… Build Competency and Confidence Develop, improve, and sustain competent & confident use of innovations. Change Organizations and Systems Create and sustain hospitable organizational and systems environments for effective instructional and educational services Provide Leadership that Matches the Challenge (c) Dean Fixsen and Karen Blase, 2008 19

Core Implementation Components Improved Child, Family, Adult Outcomes Why: EBP/Program/Initiative/Framework for Effective Practices & System Change What: Core Implementation Components Professional Development/ Professional Learning Staff capacity to support children/families with the selected practices Institutional capacity to support staff in implementing practices with fidelity Competency Drivers Organization Drivers How: There are two categories of Implementation Drivers: Competency and Organization. When these core components are in place they provide the support to a successful implementation that will be sustained. Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention to benefit students. Competency Drivers include: Selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective educational services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Capacity to provide direction, vision, & the “right” leadership approach © Fixsen & Blase, 2008

Improved Child, Family, Adult Outcomes EBP/Program/Initiative/Framework for Effective Practices & System Change Performance Assessment Implementation Drivers = Infrastructure Coaching Training Competency Drivers There are three categories of Implementation Drivers: Competency, Organization, and Leadership When these core components are in place they provide the support to establish and maintain a successful implementation of an evidence-based practice or other innovation. Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention to benefit students. Competency Drivers include: Selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective educational services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Implementation Lens Selection © Fixsen & Blase, 2008 21

Staff Training & Coaching    OUTCOMES (% of Participants who Demonstrate Knowledge, Demonstrate new Skills in a Training Setting, and Use new Skills in the Classroom) TRAINING COMPONENTS Knowledge Skill Demonstration Use in the Classroom Theory and Discussion 10% 5% 0% ..+Demonstration in Training 30% 20% …+ Practice & Feedback in Training 60% …+ Coaching in Classroom 95% The 2002 meta-analysis of training and coaching data by Joyce and Showers makes a compelling case for the need for skillful coaching. Only when training was accompanied by coaching in the service setting – in this case a classroom, was there substantial implementation in the practice setting. These findings move supervision from systems that monitor units of service, react to crises and advise around case specifics to active coaching systems that monitor adherence to evidence-based practices, are purposeful in developing practitioner skills and offer support in trying out new approaches during that “awkward stage” just after training.   Joyce and Showers, 2002 22

Sobering Observations "All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling Business Expert © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Horner, George Sugai, 2008 (c) Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 23 23

Implementation Drivers = Infrastructure Improved Child, Family, Adult Outcomes EBP/Program/Initiative/Framework for Effective Practices & System Change Implementation Drivers = Infrastructure Performance Assessment Coaching Systems Intervention Facilitative Administration Training Competency Drivers Organization Drivers There are three categories of Implementation Drivers: Competency, Organization, and Leadership When these core components are in place they provide the support to establish and maintain a successful implementation of an evidence-based practice or other innovation. Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention to benefit students. Competency Drivers include: Selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective educational services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Decision Support Data System Selection Graphics by Steve Goodman,2009 © Fixsen & Blase, 2008 24

System Stability EXISTING SYSTEM Effective Innovations are Changed to Fit the System Or Operate in the Shadows (The Ghost System)

Effective System Change EXISTING SYSTEM Effective Innovations are Changed to Fit the System Or Operate in the Shadows (Ghost System) EXISTING SYSTEM IS CHANGED TO SUPPORT THE EFFECTIVENESS OF THE INNOVATION (Host System) EFFECTIVE INNOVATION

Implementation Drivers = Infrastructure Improved Child, Family, Adult Outcomes EBP/Program/Initiative/Framework for Effective Practices & System Change Implementation Drivers = Infrastructure Performance Assessment Coaching Systems Intervention Facilitative Administration Training Competency Drivers Organization Drivers There are three categories of Implementation Drivers: Competency, Organization, and Leadership When these seven core components are in place and guided by appropriate leadership, they provide the support to establish and maintain a successful implementation of an evidence-based practice or other innovation. Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention to benefit students. Competency Drivers include: Selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective educational services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Decision Support Data System Selection Leadership Adaptive Technical © Fixsen & Blase, 2008 27

Types of Leadership Needed Different strategies for different challenges Technical Leadership Adaptive Leadership According to Ron Heifitz and his colleagues, one of the biggest mistakes “leaders” make is to incorrectly identify the type of challenge they are facing Using technical approaches for adaptive issues

The Adaptive Work of the Leader Get on the Balcony Identify the Adaptive Challenge Regulate Distress Maintain Disciplined Attention Give the Work Back to the People Protect All Voices R. Heifetz and D. Laurie: The Work of Leadership. Harvard Business Review, 1998. In adaptive work

Implementation Drivers = Infrastructure Improved Child, Family, Adult Outcomes EBP/Program/Initiative/Framework for Effective Practices & System Change Implementation Drivers = Infrastructure Performance Assessment Coaching Systems Intervention Facilitative Administration Training Competency Drivers Organization Drivers There are three categories of Implementation Drivers: Competency, Organization, and Leadership When these seven core components are in place and guided by appropriate leadership, they provide the support to establish and maintain a successful implementation of an evidence-based practice or other innovation. Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention to benefit students. Competency Drivers include: Selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective educational services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Integrated & Compensatory Decision Support Data System Selection Leadership Adaptive Technical © Fixsen & Blase, 2008 30

Implementation Frameworks Multi-dimensional, Fully integrated Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles (c) Dean Fixsen and Karen Blase, 2008 31

Stages of Implementation Implementation Takes Time 2 – 4 Years EXPLORATION INSTALLATION INITIAL IMPLEMENTATION FULL IMPLEMENTATION Stages of Implementation

Stages of Implementation You don’t get to skip any! If you make a judgment about “effectiveness” too early you might be making a mistake. Activities need to match the stage. Different stages for multiple initiatives “Where are we now with this initiative?” Stages will need to be “revisited” – important! New providers, new communities, new Ministers, new government officials, new families

Implementation Science Letting it happen Recipients are accountable Helping it happen Making it happen Implementation teams are accountable Based on Hall & Hord, 1987, Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004 (c) Dean Fixsen and Karen Blase, 2008 (c) Dean Fixsen and Karen Blase, 2008 34 34

Implementation Frameworks Multi-dimensional, Fully integrated Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles (c) Dean Fixsen and Karen Blase, 2008 35

Implementation Team(s) A group that knows the innovation very well (formal and practice knowledge) A group that knows implementation very well (formal and practice knowledge) A group that knows improvement cycles to make intervention and implementation methods more effective and efficient over time An advantage of having a well organized and persistent approach to implementation of evidence-based practices and programs may be that the purveyor can accumulate knowledge over time (Fixsen & Blase, 1993; Fixsen, Phillips, & Wolf, 1978; Winter & Szulanski, 2001). Each attempted implementation of the program reveals barriers that need to be overcome and their (eventual) solutions. Problems encountered later on may be preventable with different actions earlier in the implementation process. The Toyota Supplier and Support Center (TSSC) is a purveyor of the Toyota Production Systems for manufacturing automobiles. MST Services, Inc. is the purveyor of the Multisystemic Therapy (MST) program for serious and chronic juvenile offenders. These are clear-cut examples of purveyors and each has a set of activities designed to help new organizations ("implementation sites") implement their respective programs. In other cases, the "purveyor" is not so readily identified nor are the activities well described. Slavin, R. E., & Madden, N. A. (1999). Disseminating Success for All: Lessons for Policy and Practice (No. 30). Baltimore, MD: Center for Research on the Education of Students Placed at Risk (CRESPAR)- Johns Hopkins University. Honig, M. I. (2004). The new middle management: Intermediary organizations in education policy implementation. Educational Evaluation and Policy Analysis, 26(1), 65-87. Stetler, C. B., Legro, M. W., Rycroft-Malone, J., Bowman, C., Curran, G., Guihan, M., et al. (2006). Role of "external facilitation" in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration. Implementation Science, 1(23). Findings suggest that facilitation, within an implementation study initiated by a central change agency, is a deliberate and valued process of interactive problem solving and support that occurs in the context of a recognized need for improvement and a supportive interpersonal relationship. Harvey, G. (2002). Getting evidence into practice: the role and function of facilitation. Journal of Advanced Nursing, 37(6), 577-588. (c) Dean Fixsen and Karen Blase, 2008 36 36

Simultaneous, Multi-Level Interventions Implementation Teams Simultaneous, Multi-Level Interventions Agencies and Local Partnerships Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) Practitioners County and Local Context Service Systems Implementation Teams Purveyors also quickly learn that the Influence Factors are critical to the success of the implementation effort and over time take on a very active, simultaneous and multi-level intervention role to help increase the likelihood that such meta-contingencies as funding, licensing, referral mechanisms, regulations, and reporting requirements are aligned to support the new way of work National Context (c) Dean Fixsen and Karen Blase, 2004

Implementation Team Implementation Teams Prepare Communities Prepare staff & administrators Work with TA & Researchers Assure Implementation Prepare Agency & State Systems Assure Benefits Blase, K. A., Fixsen, D. L., & Phillips, E. L. (1984). Residential treatment for troubled children: Developing service delivery systems. In S. C. Paine, G. T. Bellamy & B. Wilcox (Eds.), Human services that work: From innovation to standard practice (pp. 149-165). Baltimore, MD: Paul H. Brookes Publishing. Slavin, R. E., & Madden, N. A. (1999). Disseminating Success for All: Lessons for Policy and Practice (No. 30). Baltimore, MD: Center for Research on the Education of Students Placed at Risk (CRESPAR)- Johns Hopkins University. Leonard-Barton, D., & Kraus, W. A. (1985). Implementing new technology. Harvard Business Review, 6, 102-110. Parents & Stakeholders Create Readiness (c) Dean Fixsen and Karen Blase, 2008 38 38 38

Implementation Teams At multiple levels Integrated and Interlocking Agency Teams Community Teams State Team Focus is on Supporting Quality Implementation Drivers Fidelity & Outcomes, Alignment (funding and policy) Problem-Solving and Sustainability

Implementation Frameworks Multi-dimensional, Fully integrated Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles (c) Dean Fixsen and Karen Blase, 2008 40

PDSA Cycles Shewhart (1924); Deming (1948); Six-Sigma (1990) Plan – Develop specific things to do Do – Do them (make sure) Study – See what happens Act – Make adjustments Cycle – Do over and over again until the goal is reached (again) Shewhart, W. A. (1931). Economic control of quality of manufactured product. New York: D. Van Nostrand Co. Deming, W. E. (1986). Out of the Crisis: MIT Press. (c) Dean Fixsen and Karen Blase, 2008 (c) Dean Fixsen and Karen Blase, 2008 41 41

What Do They Have in Common? Policy to Practice to Policy Cycles Transformation Zones Rapid Cycle Teams Problem-solving Practice Improvement Usability Testing

Improvement Cycles Policy to Practice to Policy Cycles Transformation Zones Rapid Cycle Teams Problem-solving Practice Improvement Usability Testing

Improvement Cycle Uses Why Policy-Practice Feedback Cycles? Because the Current System is perfectly designed to get the current results! And Systems Can Trump Programs!

System Reinvention Executive Management Team System Change “External” System Change Support Executive Management Team Adaptive Challenges Duplication Fragmentation Hiring criteria Salaries Credentialing Licensing Time/ scheduling Union contracts Contract methods Laws Policy Enabled Practice Practice Informed Policy System Change Implementation Team Practitioners Innovations Benefits (c) Dean Fixsen and Karen Blase, 2004 (c) Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 45 45

FORM SUPPORTS FUNCTION System Alignment Federal Departments Implementation Teams State Department Communities, Regions ALIGNMENT Provider Agencies Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation Research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Chao, S. (Ed.). (2007). The state of quality improvement and implementation research: Expert views workshop summary. Washington, D.C.: Institute of Medicine of the National Academies: The National Academies Press. Change is difficult, resistant, and often feels wrong. Inducing major change in large organizations is much more difficult than simple behavioral changes because organizations themselves are problematic. Additionally, most organization designs are outdated and do not reflect current environments, requiring more comprehensive organizational change. Freund, A. M., & Baltes, P. B. (2000). The orchestration of selection, optimization and compensation: An action - theoretical conceptualization of a theory of developmental regulation. In W. J. Perrig & A. Grob (Eds.), Control of human behavior, mental processes, and consciousness (pp. 35-58). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. Selection, optimization, compensation (SOC) theory conceptualizes the processes of adaptive development for individuals, organizations, and systems. Selection refers to narrowing the range of alternative domains of functioning or goals from the pool of available options. Optimization involves the acquisition and coordination of the means (resources) required for goal attainment. Compensation refers to managing loss or decline in goal-relevant means (resources) through substitution of means or use of external aids. Practitioners Effective Practices FORM SUPPORTS FUNCTION (c) Dean Fixsen and Karen Blase, 2008 (c) Dean Fixsen and Karen Blase, 2008 46 46 46

Implementation Frameworks Multi-dimensional, Fully integrated Implementation Drivers Implementation Stages Implementation Teams Improvement Cycles (c) Dean Fixsen and Karen Blase, 2008 47

What Will It Take to Effectively Support Children With Disabilities and Their Families? Science Related to…. Interventions and Programs Implementation, Organization, and Systems Change Funded Infrastructure (Drivers) Informed & engaged consumers & communities Linked Teams Skillful Purveyors & Intermediaries Competent Providers Supportive Organizations Engaged communities and stakeholders Hospitable & Aligned Systems Leadership at All Levels

Keep the “Big Picture” in Mind

Implementation Science Implementation Research: A Synthesis of the Literature Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: http://www.fpg.unc.edu/~nirn/resources/detail.cfm?resourceID=31   50

For More Information Karen A. Blase, Ph.D. Dean L. Fixsen, Ph.D. 919-966-9050 Karen.Blase@unc.edu Dean L. Fixsen, Ph.D. 919-966-3892 Dean.Fixsen@unc.edu At the Frank Porter Graham Child Development Institute University of North Carolina Chapel Hill, NC www.scalingup.org http://nirn.fpg.unc.edu/ http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/  (c) Dean Fixsen and Karen Blase, 2009

For More Information Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/  To order the monograph go to: https://fmhi.pro-copy.com/  (c) Dean Fixsen and Karen Blase, 2009