Working Smarter: The implementation journey to improve student outcomes Michelle A. Duda, Ph.D., BCBA, Dean L. Fixsen, Ph.D. & Karen A. Blase Ph.D., Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University of North Carolina – Chapel Hill

Implementation Fluency Part 2

Science to Service GAP SERVICE SCIENCE IMPLEMENTATION

Science to Service Science to Service Gap Implementation Gap What is known is not what is adopted to help students, families, and communities Implementation Gap What is adopted is not used with fidelity and good outcomes for consumers. What is used with fidelity is not sustained for a useful period of time. What is sustained is not used on a scale sufficient to impact societal outcomes.

Making use of Science Letting it happen Helping it happen Recipients are accountable Helping it happen Making it happen Implementation teams are accountable Based on Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004

What is Implementation? What do we mean by implementation? A specified set of purposeful activities at the practice, program, and system level designed to put into place a program or intervention of known dimensions with fidelity.

In Other Words… It’s all about changing the behavior of well intentioned people including practitioners, providers, community stakeholders, policy makers and funders!

Ineffective Methods Excellent evidence for what does not work Implementation by mandate does not work Implementation by “following the money” does not work Implementation without changing supporting roles and functions does not work Paul Nutt (2002). Why Decisions Fail

Ineffective Methods Excellent experimental evidence for what does not work Diffusion/dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines) Training alone, no matter how well done, does not lead to successful implementation

Implement Innovations IMPLEMENTATION Effective NOT Effective Student Benefits Effective INTERVENTION Placebo NOT Effective PLACEBO: Something of no intrinsic remedial value that is used to appease or reassure another

Good outcomes for consumers What Works Effective intervention practices + Effective implementation practices = Good outcomes for consumers

Common Components of Fidelity Measures Adherence Dosage/Exposure Quality of Program Delivery

Fidelity Data Collection Methods Direct Observation Practitioner Self-report Consumer Self-report

Why are programs not implemented with fidelity? Lack of teacher training, Lack of required materials Use of some (but not all) of the required lessons and teaching strategies l Limited time _____________________________ Lack of funding Inadequate infrastructure Decentralized decision making Lack of administrative support

Why are programs adapted? Inadequate training/lack of understanding of program’s underlying theory Barriers (time, money, resources, accessing target population) Programs that are not user-friendly Lack of perceived efficacy, relevance, acceptance Brian Bumbarger, Prevention Research Center

Strategies for Maintaining Fidelity Creating awareness around the importance of maintaining fidelity Improving practitioner understanding of program’s underlying theory and rationale Developing sustainable infrastructure and processes for monitoring implementation and fidelity Monitoring fidelity continually

Evidence-Based Movement The “evidence-based movement” is an international experiment to make better use of research findings in typical service settings. The purpose is to produce greater benefits to students and society.

EBPs & Implementation Core intervention components Clearly described (who/what) Practical measure of fidelity Fully operationalized (do/say) Field tested (recursive revision) Contextualized (org./systems fit) Effective (worth the effort) Nielsen, J. (2000). Why You Only Need to Test With 5 Users. Retrieved April 22, 2007, from http://www.useit.com/alertbox/20000319.html Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York: John Wiley & Sons. Frick, T., Elder, M., Hebb, C., Wang, Y., & Yoon, S. (2006). Adaptive usability evaluation of complex web sites: How many tasks? Unpublished manuscript, Indiana University, W.W. Wright Education 2276, 201 N. Rose Ave., Bloomington, IN 47405-1006. Allen, B. L. (1996). Information tasks: Toward a user-centered approach to information systems. New York: Academic Press.

EBPs & Implementation The usability of a program/practice has little to do with the quality or weight of the evidence regarding that program Evidence on intervention effectiveness for specific populations helps us choose what to implement Evidence on the effectiveness of the intervention does not help implement the program or practice successfully

EBPs & Implementation From an implementation perspective, what do we need to know about innovations such as evidence-based programs? Successful and sustainable implementation of evidence-based programs always requires organization and systems change.

Implementation Frameworks In order to ensure that an EBP maintains and sustains producing benefits to students over time, it is critical to ensure that the system is both multi-dimensional and fully integrated through : Implementation Stages (PART 3) Implementation Drivers (PART 4) Improvement Cycles (PART 5)

Dean L. Fixsen, Ph.D. 919-966-3892 fixsen@mail.fpg.unc.edu For More Information Michelle A. Duda, Ph.D. 919-636-0843 duda@mail.fpg.unc.edu Dean L. Fixsen, Ph.D. 919-966-3892 fixsen@mail.fpg.unc.edu At the Frank Porter Graham Child Development Institute University of North Carolina Chapel Hill, NC www.scalingup.org http://nirn.fpg.unc.edu/ http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/ 

For More Information Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/  To order the monograph go to: https://fmhi.pro-copy.com/