Presentation on theme: "EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH: THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS Maria Scannapieco, Ph.D., Professor & Director Center."— Presentation transcript:
EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH: THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS Maria Scannapieco, Ph.D., Professor & Director Center for Child Welfare Mountains & Plains Child Welfare Implementation Center 25 April 2011
Overview Introduction to Implementation Centers Basic understanding of Implementation Science Reflections on lessons learned in systems change and implications for states
Implementation Centers and National Resource Centers 4 Improve child welfare administration and practice Meet Federal performance standards and implement effective programs Pursue sustainable and positive systems change Improve outcomes for children, youth & families Funded by the Children’s Bureau to provide States and Territories, Tribes, and Tribal Consortia individualized training and technical assistance (T/TA) to:
Implementation Centers: Filling the Gap 5 States and Tribes are sometimes without the resources necessary to implement comprehensive strategic plans National Resource Centers have limited resources to provide intensive, long-term TA Implementation Centers provide in-depth and long-term consultation and peer networking opportunities to States and Tribes
Project Partner Roles Building Capacity to Implement Sustainable Systems Change Implementation CenterChild Welfare Agency Conduct organizational and readiness assessment to identify potential barriers to implementation Develop and implement strategies to address organizational barriers and improve readiness for implementation Guide planning processDevelop project plan Provide and coordinate technical assistance resources to support organizational change and facilitate implementation Provide and coordinate organizational/ human resources to manage change and support implementation Provide technical assistance to promote best practices in organizational and child welfare systems change Utilize best practices to align people, process, structure, measurement/rewards, and technology around the desired systems change Facilitate implementationImplement plan Develop methodology (e.g. benchmarks, tools) to monitor implementation progress Collect project data; use data reports to monitor implementation and inform /refine implementation strategies Evaluate project outcomesUse findings and lessons learned to inform future implementation efforts
Who are the Implementation Centers? 8 Implementation CenterOrganization Northeast & Caribbean Implementation Center (NCIC) University of Southern Maine, Muskie School of Public Service Atlantic Coast Child Welfare Implementation Center (ACCWIC) University of Maryland School of Social Work Midwest Child Welfare Implementation Center (MCWIC) University of Nebraska-Lincoln, Center on Children, Families & the Law Mountains & Plains Child Welfare Implementation Center (MPCWIC) University of Texas at Arlington, partnering with the University of Denver and The Native American Training Institute (NATI) Western & Pacific Child Welfare Implementation Center (WPIC) American Institutes for Research, partnering with National Indian Child Welfare Association, Center for the Study of Social Policy, National Technical Assistance Center for Children’s Mental Health at Georgetown University, and the Florida Mental Health Institute at the University of South Florida
Mountains and Plains Child Welfare Implementation Center MPCWIC Geographical Service Area Region 6: Arkansas, Louisiana, New Mexico, Oklahoma, Texas Region 8: Colorado, Montana, North Dakota, South Dakota, Utah, Wyoming
MPCWIC Projects The Skun-eyah (Garden) Project is a collaborative project to be implemented by two tribes, Three Affiliated Tribes (Mandan, Hidatsa and Arikara Nations) and Turtle Mountain Band of Chippewa The Osage Nation implementation project goal is three-fold: develop a business mapping model, develop and articulate a culturally based family centered practice model, and create a decision support data system to help facilitate the first two Colorado Department of Human Services, Division of Child Welfare– Child Welfare Practice Reform. This three-year project will explore, define and implement a Colorado practice model, inclusive of specific practice strategies, methods and tools to improve outcomes for children and families. Denver, Colorado Developing a New Mexico Child Welfare Practice Modelwill be focused on the development and implementation of a clearly articulated practice framework, inclusive of vision, mission, values and operating principles, to guide all of the change initiatives underway in the State
Change Management Model Awareness of the need for change Desire to participate and support the change Knowledge of how to change Ability to implement the required skills/behaviors Reinforcement to sustain the change
Competencies for Leading Change Creativity and innovation External awareness Flexibility Strategic Thinking Vision
Implementation is not a Hobby! There is a body of research around effective implementation. Child welfare is slow to apply the science. Implementation Centers are utilizing methodology to support project work with states and tribes.
What do we know? Implementation Science National Implementation Research Network (NIRN) Phases of Change John Kotter, Harvard Business School Managing Change ADKAR Model (Awareness, Desire, Knowledge, Ability and Reinforcement)
Implementation Frameworks Brief overview of the science of implementation Practice, program and systems change through… Multi-dimensional, fully integrated use of Implementation Stages Implementation Drivers Implementation Teams
Applying Frameworks The value of frameworks is To promote the ability to generalize beyond the immediate project or initiative To enhance communication among partners (e.g. better understanding of one another) To more easily share and apply improvements To increase the relevance of the “lessons learned”
Building the Implementation Platform Brief overview of the science of implementation Multi-dimensional, fully integrated Implementation Stages Implementation Drivers Transformation Zones
Science to Service SCIENCE SERVICE GAP IMPLEMENTATION
What Do We Mean by Implementation ? A specified set of activities designed to put into practice a policy, activity, or program of known dimensions Implementation processes are purposeful and defined in sufficient detail such that independent observers can detect the presence and strength of these “specified activities”
Overall Challenge Science to Service Gap What is known often is not what is adopted to help children, families, and caregivers Implementation Gap There are not clear pathways to implementation What is adopted often is not used with fidelity and good effect What is implemented often disappears with time and staff turnover
From the Synthesis of the Literature We Know That… Implementation issues are common across widely diverse domains Human service prevention and intervention (e.g. child welfare, substance abuse, mental health, violence prevention, education) Advanced manufacturing technologies Research-based clinical guidelines Engineering (e.g. bridge maintenance) Hotel management National franchise operations Cancer prevention and treatment
Positive Intervention Outcomes ≠ Implementation Implementation has not been achieved by doing more or better research on interventions or on curricula The usability of a program or practice has nothing to do with the weight of the evidence regarding that program –“Evidence” on effectiveness helps you select what to implement for whom –“Evidence” on these outcomes does not help you implement the program or practice
Insufficient Methods Implementation by laws/ compliance by itself does not work Implementation by “following the money” by itself does not work Implementation without changing supporting roles and functions does not work Fixsen, Naoom, Blase, Friedman, Wallace, 2005
Insufficient Methods Diffusion/dissemination of information by itself does not lead to successful implementation Training alone, no matter how well done, does not lead to successful implementation Fixsen, Naoom, Blase, Friedman, Wallace, 2005 So what does work?
What Works EffectiveNOT Effective Effective NOT Effective IMPLEMENTATION – The HOW INTERVENTION The WHAT Performance Implementation (High Fidelity) Paper Implementation Procedure Implementation (Low Fidelity) Getting Here and Staying Here is the Challenge!
Child Welfare Practice Classification System Well-Supported, Efficacious Practice Evidenced Based Practice (EBP) Promising and Acceptable Practice Best Practices Innovative or Novel Practice Concerning Practice
Stages of Implementation ExplorationDesign/Installation Initial Implementation Full Implementation Sustainability Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network
Stages of Implementation Core Activities Exploration Conduct organizational assessment Garner leadership and stakeholder support Review literature Solicit multiple perspectives on needs and opportunities Establish baseline performance Develop intervention model Design/Installation Develop plans: Implementation Communication Change management Define goals and benchmarks Engage stakeholders Conduct readiness assessment Align organization systems/supports Initial Implementation Build staff competency Communicate project status Monitor fidelity of implementation Monitor implementation climate Identify and address barriers to implementation Provide performance feedback to staff, management, and stakeholders Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network
Stages/Activities Full Implementation Apply lessons learned from initial implementation to full operation Track fidelity through quality assurance and performance evaluation data Solicit feedback from multiple stakeholders and consumers Evaluate impact on child and family outcomes Innovation Adapt or adjust intervention model to reflect lessons learned from stakeholder feedback, tracking, evaluation Communicate changes and rationale Re-build competency around modified model Sustainability Establish long term funding sources Align ongoing quality assurance and performance measurement with model Promote visibility of new practice and successful outcomes Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network
System Change Demonstrations or “pilots” are a place to start for innovations (“it’s possible!”) Don’t usually lead to sustainable service and system change Random acts of innovation Person and passion dependent Can “ghost” system its way to success Executed by the “extraordinary” No replicable implementation infrastructure
Transformation Zone Focus on innovations And implementation infrastructure development And organizational change And systems change processes
Transformation Zone Use Innovations Develop Implementation Infrastructure Change System
Transformation Zone A “vertical slice” of the service system (from the front-line to the Capitol) The “slice” is small enough to be manageable The “slice” is large enough to include all aspects of the system (court, case work, service system, urban, rural, frontier, diverse communities) The “slice” is large enough to “disturb the system” – a “ghost” system won’t work.
Transformation Zone Transformation Zones provide opportunities to… Manage the risks (most innovations don’t work at first) and experience intended and unintended outcomes Limit the damage (quick detection, recovery, repair) Document “what works” Consider the implications of scaling-up
TA Outcomes Evaluation Questions: Research QuestionIndicatorProcess Measures To what extent do implementation project stakeholders view MPCWIC technical assistance as having increased their State/Tribal capacity to meet project goals? Perceptions of increased state/tribal implementation capacity Implementation Capacity Assessment Key Stakeholder Interviews What components of NIRN are applicable in implementation of child welfare system change projects? Stages/Drivers alignment with implementation success Implementation Process Analysis Global Attainment Scaling Driver Analysis
TA Outcomes Evaluation Questions: Research QuestionIndicatorProcess Measures To what extent do implementation project stakeholders view MPCWIC technical assistance as helpful in addressing state/tribal systems barriers? Perceptions of increased state/tribal implementation capacity to address systems barriers Key Stakeholder Interviews To what extent do implementation project stakeholders view the NIRN framework as helpful in addressing system barriers? Perceptions of Stages/Drivers installation related to implementation success Key Stakeholder Interviews Implementation Capacity Assessment Implementation Process Analysis
MPCWIC Outcome Evaluation Individual Implementation Project Evaluation Do State and Tribal systems improve with respect to the issue addressed in their individual implementation projects? Do State and Tribal systems make significant advancements toward the development of successful practice models? How do State and Tribal characteristics or organizational factors contribute to the development of successful practice models? What is the impact of the implementation project activities and integrations of the practice model on State or Tribal CSFR and PIP outcomes? Individually, do State and Tribal partners accomplish the goals set forth in their multi-year strategic plans for sustainable system’s change? Are State and Tribal partners successfully able to use practice model frameworks with fidelity to guide daily practice?
MPCWIC Outcome Evaluation Cross- Implementation Project Outcomes As a group, do State and Tribal partners accomplish goals set forth in their multi-year strategic plans for sustainable systems change? Do organizations develop successful practice models? What organizational factors are associated with successful practice models? Do project type and organizational characteristics differentially affect outcomes? What is the relationship between TA intensity and type and overall outcomes, particularily those related to culture, climate leadership, supervision and practice?
Summary Stage related activities prepare the system for a successful change process Competence needs to be developed and sustained Selection, training, coaching, fidelity measures help change and support new practitioner behavior and skills Organizations and systems need to change Data systems need to be used to make decisions Facilitative administrative practices & systems interventions create hospitable environments Policy enables new practice but practice needs to inform policy