Using Observation to Enhance Supervision CIMH Symposium Supervisor Track Oakland, California April 27, 2012

FFT is an Evidence-Based Practice Four decades of research support establishing the efficacy of FFT with behavior problem youth Recidivism/arrest, including violent felony crimes Drug use Out of home placement Prevention of arrest in siblings of referred youth Reduction in youth, mother, father interpersonal distress/somatic complaints Improvement in family functioning Significant cost effectiveness (up to $14.87 return for each dollar spent) Findings replicated across research teams, with therapists with diverse clinical experience, and with culturally diverse populations

Science to Practice Challenge in dissemination is to ensure that therapists implement FFT with the same rigor and fidelity established in controlled trials

Fidelity and Outcomes: State of Washington Implementation 38%* reduction in felony recidivism $10.67 return for each $1 invested $2100 per family cost to implement * Statistically significant outcome as compared to control condition Wash State Institute for Public Policy, 2005

The Effect of Supervised Experience: Felony Recidivism by Level of Therapist Competence The data for this slide was obtained from the Statewide Washington State FFT Project. The case experience was clustered into three levels: 1-7, 8-14, and 15-21. The high competence therapists are the competent and High competent group combined while the low competent group are the “borderline and non competent” groups combined. Case Experience Aos et al., 2004 5

Summary of Washington Findings Only half the therapists were actually implementing FFT in a competent manner Reductions in felony recidivism were evident only for therapists implementing the model competently For the subset of competent FFT therapists, the financial benefits of the dissemination were estimated at $7.50 for each dollar of program cost (Aos et al., 2004)

Modifications to Training and Dissemination System Intensive engagement of stakeholders and buy-in during site set-up process Revisions to the Client Services System Web-based tracking system Implementation of new procedures for training and supervising therapists Increased accountability for practice using data-informed practice guidelines

What Works to Reduce Crime?   Juvenile Offenders Change in Crime (# of EB Studies) Benefits Minus Costs (per-person, life cycle) Probability: you lost $ Functional Family Therapy -18% (7) $32,021 (<1%) Multisystemic Therapy -13% (10) $18,120 (<1%) Family Integrated Transitions -10% (1) $29,721 (~5%) Multi-Dimensional Treatment Foster Care -18% (3) $64,486 (<1%) Adapted from Aos (2010)

Return-on-Investment of Juvenile Offender Programs on Crime Reduction (WSIPP)   Juvenile Program Program Cost Per Program Participant Victimizations Avoided Per Program Participant Taxpayer Benefits Per Program Participant Victim Benefits Per Program Participant Per $/1000 Program Cost State Percent of Benefits Multi-Dimensional Treatment Foster Care $7418 0.10 50% $24,068 Family Integrated Transitions $10795 0.04 $13,050 Coordination of Services $379 0.19 45% $2,135 Functional Family Therapy $3134 0.22 $20,623 Multi-Systemic Therapy $7076 0.05 $11,027

Next Steps Is FFT effective in real world settings? Moving from efficacy to effectiveness research What procedures are necessary to achieve the most potent effects? Observation-based supervision has been critical in the development of clinical practices and in efficacy trials The costs associated with supervision based on a direct review of therapy session recordings are often viewed as prohibitive . Does observation-based supervision influence therapist competence and subsequent treatment outcomes? The sustainability of community implementations must be considered in the context of empirical evidence for the procedures in relation to outcomes achieved.

Primary Objective To examine the effects of observation-based supervision (BOOST) versus the standard supervision as usual (SAU) approach currently used by FFT LLC on the therapists’ ability to facilitate changes in the family. BOOST will examine the following outcomes: adolescent drug use conduct/delinquent behaviors. engaging youth/families in treatment enhancing therapist competence 

Engagement-Retention Mediating Variables Outcome Variables Engagement-Retention in Treatment AIM 2 AIM 4 Therapist Competence AIM 3a Supervision Condition AIM 4 AIM 3b AIM 4 Family Functioning Adolescent Substance Use AIM 1a, 1b Adolescent Conduct / Delinquency Conceptual Model of Hypothesized Relationships

Random Assignment of FFT Supervision Teams Supervision as Usual 8 Teams - 3 Therapists Each (n=24) Observation-Based Supervision 8 Teams – 3 Therapists Each (n=24) Family Referral (n=288) Screening and Informed Consent Baseline Assessment Functional Family Therapy Follow-Up Assessments: 5-and 12-months after Baseline

Implementation at Each Site Participants will be 18 adolescents/families per team FFT teams will include 3 therapists (6 families per therapist) Duration of study at each site Month 1: Start Up; Therapist Consent; Randomization of FFT teams Month 2-8: Enrollment of family participants Months 2 to 12: Clinical services * Months 6-12: Completion of 5-month assessment Months 14-20: Completion of 12-month assessment *Supervisor and Therapist activities completed. Therapists in SAU eligible to participate in BOOST training

Features of the Study One of the only formal examinations of dissemination outcomes in evidence-based practices for youth with disruptive behavior problems First study of supervision procedures! Inclusion of a diverse sample of family and therapist participants across multiple community agencies Tracking treatment outcomes and post-treatment outcomes Protocol is fully bi-lingual (including computerized assessment procedures) BOOST procedures will be transported into participating agencies to enhance sustainability after study Includes procedures to train therapists and site supervisors in SAU after clinical phase of study is completed at each site

Cautions Intensity of review can be overwhelming Rule of 5 to 1 Try not to blow smoke Specificity and brevity Matching to therapist and family Let therapist and team make comments and recommendations