Measuring and Improving Fidelity to Evidence-Based Practices Our Obligation to Effective Service Delivery and Supervision Kimberly Gentry Sperber, Ph.D.

Workshop Objectives Review literature on fidelity Identify opportunities for assessing fidelity Identify resources for monitoring and improving fidelity Address barriers to monitoring fidelity

Objective 1 What Does Science Say About Fidelity?

Latessa, Cullen, and Gendreau (2002) Article notes 4 common failures of correctional programs: –Failure to use research in designing programs –Failure to follow appropriate assessment and classification practices –Failure to use effective treatment models –Failure to evaluate what we do

The Role of Fidelity Lowenkamp and Latessa (2005) Examined data from 38 residential correctional programs for adults Looked at relationship between program fidelity and program effectiveness. Program fidelity was assessed using the CPAI. Found significant correlation between fidelity and effectiveness –CPAI scores correlated to reincarceration

Lowenkamp and Latessa Findings Continued Differences in recidivism rates based on CPAI scores: –Scores of 0-49% demonstrated 1.7% reduction compared to comparison group. –Scores of 50-59% demonstrated 8.1% reduction. –Scores of 60-69% demonstrated 22% reduction.

CPAI Data Continued Holsinger (1999) –Examined data from Adolescent Community Correctional Facilities in Ohio –Looked at relationship between program fidelity and program effectiveness. –Program fidelity was assessed using the CPAI. –Outcome measures examined included any court contact, felony or misdemeanor, felony, personal offense, and commitment to a secure facility

CPAI Data Continued Total composite score significantly correlated with all outcome measures. Each individual domain of the CPAI also significantly correlated with all of the outcomes –Program Implementation –Client Assessment –Program Characteristics –Staff Quality –Evaluation

More Fidelity Research Landenberger and Lipsey (2005) –Brand of CBT didn’t matter but quality of implementation did. –Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers. Sexton (2001) –Direct linear relationship between staff competence and recidivism reductions. Schoenwald et al. (2003) –Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients. Henggeler et al. (2002) –Supervisors’ expertise in the model predicted therapist adherence to the model.

More Fidelity Research Cont’d. McHugo et al. (1999) found that Assertive Community Treatment (ACT) teams with higher fidelity showed greater reductions in substance use, higher rates of remission, and fewer hospitalizations for dual diagnosis clients. Kirigin et al. (1982) found that higher fidelity among staff was associated with greater reductions in delinquency. Bruns et al. (2005) compared high fidelity Wraparound sites to low fidelity sites and found high fidelity sites to result in improved social/academic functioning of children and lower restrictiveness of placements. Drake et al. (1996) found that the site with higher fidelity to the Supported Employment Model had better employment outcomes for clients.

More Fidelity Research Cont’d. Schoenwald and Chapman (2007) –A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatment –A 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment.

Washington State Example (Barnoski, 2004) For each program (FFT and ART), an equivalent comparison/control group was created Felony recidivism rates were calculated for each of three groups, for each of the programs Youth who received services from therapists deemed ‘competent’ Youth who received services from therapists deemed ‘not competent’ Youth who did not receive any services (control group)

Functional Family Therapy Results: % New Felony Results calculated using multivariate models in order to control for potential differences between groups

Washington State Study Continued When FFT was delivered competently, the program reduced felony recidivism by 38% When considering how much the program costs, substantial savings in ‘avoided crime’ were observed – particularly for the competent therapists When ART was competently delivered, felony recidivism was reduced by 24% Also resulted in substantial savings

Project Greenlight Short-term prison-based reentry program in New York –CBT Skills Training –Employment Services –Housing Services –Drug Education and Awareness –Family Counseling –Practical Skills Training –Community-Based Networks –Familiarity With Parole –Individualized Release Plans

Project Greenlight Benefits Participants received more service referrals Participants reported more contacts with community services after release Participants demonstrated significantly more familiarity with parole conditions Participants were more positive about parole

But Did It Work?

What Went Wrong? Violation of the risk principle –Ceased use of risk assessment instrument when staff deemed process too cumbersome Violation of the need principle –All offenders received same services whether needed or not Violation of the fidelity principle –Staff modified delivery of the CBT curriculum (shortened the duration, increased frequency, increased class size) Differential staff competence –Certain case managers produced worse outcomes

2010 UC Halfway House/CBCF Study in Ohio: Adherence to CBT in Groups and Changes in Recidivism

Flores, Lowenkamp, Holsinger, & Latessa, (2006) –Higher correlations among scores produced by trained staff and future incarceration –Lower correlations among scores produced by untrained staff and future incarceration –Higher correlations among agencies using the LSI-R for at least 3 years and future incarceration –What are the implications of these findings? Fidelity and the Validity of the LSI-R

Fidelity, Emotional Exhaustion, and Staff Retention Aarons et al. (2009): –Staff who received ongoing consultation for fidelity in the EBP group had significantly lower turnover than staff receiving ongoing consultation in the TAU group. –Staff in EBP also demonstrated less emotional exhaustion

What Do We Know About Fidelity? Fidelity is related to successful outcomes (i.e., reductions in recidivism, relapse, and MH instability). Poor fidelity can lead to null effects or even iatrogenic effects. Fidelity cannot be assumed Fidelity can be measured and monitored.

Relationship Between Evaluation and Treatment Effect (based on UC Halfway House and CBCF study)

NPC Research on Drug Courts

Objective 2 Opportunities for Assessing Fidelity

Opportunities to Monitor Fidelity Training Assessments Treatment groups Individual sessions Case Management Milieu Documentation Review Program Assessments

Ensuring Training Transfer Use of knowledge-based pre/post-tests Use of knowledge-based proficiency tests Use of skill-based rating upon completion of training Mechanism for use of data –Staff must meet certain criteria or score to be deemed competent. –Failure to meet criteria results in consequent training, supervision, etc.

Assessments Desktop Reviews –Accurate scores –Reviews of overrides –Integration with service plan/dosage Observations –Use of standardized audit sheet –Assess interviewing skills –Assess accuracy of item ratings

Treatment Groups Observation-based ratings of adherence to treatment model. –CBT: Frequency of role-plays Structure of role-plays Appropriateness of role-plays Use of behavioral reinforcers Effective use of authority and disapproval Teaches the thought-behavior chain Teaches structured skill building Follows curriculum

Individual Sessions Observation-based ratings of adherence to treatment model. –CBT: Teaches thought-behavior chain Teaches problem-solving Teaches structured skill building Conducts role-plays Appropriate use of thinking reports/homework Graduated practice Appropriate use of reinforcers

Case Management Observation-based ratings of adherence to treatment model. –CBT: Teaches thought-behavior chain Teaches problem-solving Teaches structured skill building Appropriate use of reinforcers Helps client to integrate skills learned into real world environment (e.g., employment)

Milieu Observation-based ratings of competence in core correctional practices –Focus is more on effective use of authority and disapproval and appropriate use of reinforcers and sanctions. Standardized list of behavioral indicators Structure for observing and rating staff interacting with clients in milieu Can also review incident data for trends

Documentation Review: Why Do It? Clinical Implications –Documentation is not separate from service delivery. –Did the client receive the services he/she needed? Operational Implications –Good documentation should drive decision- making. –Means of communication Risk Management Implications –If it isn’t documented, it didn’t happen. –Permanent record of what occurred in the facility Source of Staff Training Reflection of the provider and organization’s competency: –EBP –Outcome of care

Program Assessments Correctional Program Checklist (CPC) Correctional Program Assessment Inventory (CPAI) ICCA Treatment Survey

Sample Measures Percentage of groups containing role-plays Percentage of successful completers receiving appropriate dosage based on risk/needs assessment Percentage of staff achieving 4:1 ratio Percentage of groups observed where staff modeled the skill prior to having clients engage in role-play Percentage of role-plays containing practice of the correctives Percentage of role-plays that required observers to identify skill steps and report back to the group

Objective 3 Resources Required to Monitor and Improve Fidelity

Observation-Based Ratings Creation of audit sheets Schedule for conducting the reviews Staff qualified to conduct and rate the observations Time for staff to conduct observations Mechanism to record and use the data –Supervision and individual staff development –QI and training initiatives

Documentation Review Staff to conduct the review Schedule for review rotation Audit sheet Time to conduct the review Mechanism for recording and using the data –Action planning

Objective 4 Barriers to Monitoring Fidelity

Common Barriers Strength of conceptual understanding of the EBP to be measured Resources Setting priorities Understanding/skill sets required for measurement Conflicting philosophies (helper vs. evaluator) Time!

Potential Strategies Start small –For example, desk top review of assessments versus observation-based ratings Use technology to increase efficiencies –For example, videotape interactions for observation-based ratings Take the time to build expertise –Train on model –Train on evaluation methodology –Insure understanding of purpose (e.g., QI versus punishment)

Conclusions Many agencies are allocating resources to selection/implementation of EBP with no evidence that staff are adhering to the model. There is evidence that fidelity directly affects client outcomes. There is evidence that internal evaluation processes directly affect client outcomes. Therefore, agencies have an obligation to routinely assess and assure fidelity to EBP’s. Requires a formal infrastructure to routinely monitor fidelity performance.

Questions and Answers Contact Information: