1. RE-EXAMINING THE ROLE OF PROFESSIONAL DEVELOPMENT AND TRAINING EVALUATION THROUGH AN IMPLEMENTATION SCIENCE LENS MICHELLE GRAEF & ROBIN LEAKE NHSTES June 15, 2011

2. Presentation Overview  Children’s Bureau’s Child Welfare Implementation Centers  What do we mean by “implementation”?  Overview of one Implementation Science framework (NIRN)  Implications of framework for training and training evaluation

3. Implementation Centers and National Resource Centers  Improve child welfare administration and practice  Meet Federal performance standards and implement effective programs  Pursue sustainable and positive systems change  Improve outcomes for children, youth & families Funded by the Children’s Bureau in 2008 to provide States and Territories, Tribes, and Tribal Consortia individualized training and technical assistance (T/TA) to:

4.  States and Tribes are sometimes without the resources necessary to implement comprehensive strategic plans  National Resource Centers have limited resources to provide intensive, long-term technical assistance  Implementation Centers provide in-depth and long-term technical assistance and peer networking opportunities to States and Tribes Implementation Centers: Filling the Gap

6. What do we mean by “implementation”?  The process of moving an idea from concept to reality  “a specified set of activities designed to put into practice an activity or program of known dimensions” (NIRN)  Involves both:  project management (managing the tasks, resources, budget, process –technical)  change management (managing the people – adaptive)

7. In other words…

8. Science of Implementation  There is a body of research on effective implementation, mostly from research on evidenced-based practices  Child welfare systems have been slow to explore the science, often using “paper implementation”  The Implementation Centers are using several models to guide their work:  National Implementation Research Network (NIRN)  Phases of Change (John Kotter)  ADKAR (Awareness, Desire, Knowledge, Ability, Reinforcement) - Prosci

9. Selection Training Coaching Performance Assessment Systems Intervention LEADERSHIP Decision Support Data System Integrated & Compensatory COMPETENCY Drivers ORGANIZATIONAL Drivers Facilitative Administration © Fixsen & Blase, 2008 National Implementation Research Network: Implementation Drivers

10. Stages of Implementation Core Activities Exploration Conduct organizational assessment Garner leadership and stakeholder support Review literature Solicit multiple perspectives on needs and opportunities Establish baseline performance Develop intervention model Design/Installation Develop plans: Implementation Communication Change management Define goals and benchmarks Engage stakeholders Conduct readiness assessment Align organization systems/supports Initial Implementation Build staff competency Communicate project status Monitor fidelity of implementation Monitor implementation climate Identify and address barriers to implementation Provide performance feedback to staff, management, and stakeholders Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

11. Stages of Implementation Core Activities (cont’) Full Implementation Apply lessons learned from initial implementation to full operation Track fidelity through quality assurance and performance evaluation data Solicit feedback from multiple stakeholders and consumers Evaluate impact on child and family outcomes Innovation Adapt or adjust intervention model to reflect lessons learned from stakeholder feedback, tracking, evaluation Communicate changes and rationale Re-build competency around modified model Sustainability Establish long term funding sources Align ongoing quality assurance and performance measurement with model Promote visibility of new practice and successful outcomes Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

12. So does any of this really happen in child welfare agencies?  ICs have developed a common measure of our implementation process, based on NIRN drivers but includes others  Collecting data across all projects:  summarize implementation drivers that are salient,  describe implementation activities happening over life of projects, across multiple stages.

13. Intervention vs. Implementation Outcomes  Important to differentiate between intervention- level activity/outcomes and implementation-level activity/outcomes (Proctor et al, 2010).  Examples of implementation outcomes:  Acceptability (user satisfaction)  Adoption/uptake (utilization or intent to try)  Feasibility (suitability for use, practical)  Fidelity (adherence to program model)  Penetration (spread across settings)  Sustainability (level of institutionalization)

14. Implications for Role of Training  What is the role of training in the NIRN framework? How is this different/not different from the way training is usually conceptualized by CW agencies?  Training alone, no matter how well done, does not lead to successful implementation Fixsen, Naoom, Blase, Friedman, Wallace, 2005

15. Implications for Framework  How does overall professional development fit within the NIRN framework?  Why is training depicted separately from coaching in the triangle? Is there a better way to model this?

16. Implications for Role of Training Evaluation  What is the role of training evaluation in this framework?  What are the boundaries between training evaluation and implementation evaluation? Is there a distinction? Should there be?  What is the implication for the work that we do?

17. Learn More Implementation CentersWebsite Northeast & Caribbean Implementation Center (NCIC) http://ncic.muskie.usm.maine.edu Atlantic Coast Child Welfare Implementation Center (ACCWIC) www.accwic.org Midwest Child Welfare Implementation Center (MCWIC) www.mcwic.org Mountains & Plains Child Welfare Implementation Center (MPCWIC) www.mpcwic.org Western & Pacific Child Welfare Implementation Center (WPIC) www.wpicenter.org