Effectiveness of two-stage training for brief interventionists in a multi-site trial Michael P. Bogenschutz, 1 Alyssa A. Forcehimes, 1 Christine Sanchez, 1 Dennis M. Donovan, 2 Chris Dunn, John Baer, 2 Karin Wilson, 1 Raul N. Mandler, 3 Harold I. Perl, 3 Theresa B. Moyers 1 1 University of New Mexico, 2 University of Washington, 3 National Institute on Drug Abuse INTRODUCTION METHODS RESULTS CONCLUSIONS ACKNOWLEDGEMENTS The primary aim of the SMART-ED study is to contrast substance use and substance-related outcomes among patients who endorse problematic substance use during an ED visit and are randomly assigned to one of three treatment conditions: 1) minimal screening only (MSO); 2) screening, assessment, and referral to treatment (if indicated or requested) (SAR); and 3) screening, assessment, and referral (if indicated or requested) plus a brief intervention (BI) with two telephone follow-up booster sessions (BI-B). Individuals presenting to an ED who endorse problematic substance use on screening are randomized in 1:1:1 ratio to MSO vs. SAR vs. BI-B. Those assigned to the BI-B group receive a brief intervention consisting of motivational enhancement therapy (MET) adapted for use in the ED, followed by the informational pamphlet and referral if indicated or requested. The BI-B group also receive two booster telephone calls which will occur within 1 week of enrollment if possible, and in no cases later than 1 month following enrollment. The study was implemented in two waves, such that the first two sites were received training and began recruitment first, followed 6 months later by the second group of 4 sites. The booster interventionists were trained separately and serve all 6 sites. Prior to completing any training, interventionists provided informed consent and completed questionnaires including demographic information, training and experience, and knowledge of MI and other counseling approaches. The Short Understanding of Substance Abuse Scale (SUS, Humphreys et al., 1996) was used to assess their theoretical model for understanding addiction. This scale yields scores for three reliable subscales representing the Disease Model, the Psychological Model, and the Eclectic Model. Interventionists then received a 2-day training in basic Motivational Interviewing skills, conducted locally, followed 1 week later by a 2-day training in the specific counseling intervention used in the trial, conducted as part of the national protocol start- up meeting. They then completed practice sessions with consenting ED patients. Audiotapes of these sessions were reviewed by an expert rater, using the Motivational Interviewing Treatment Integrity scale (MITI, version 3.1.2) (Moyers et al., 2005) and content checklists. The MITI yields 4 summary scores as well as several calculated scores based on coding of utterances during the rated tape. To become certified, interventionists had to score at least 4.0 (“competent”) on the 5-point Global Clinician Rating from the MITI and 80% on content checklists, for 2 out of 3 sessions. Supported by the National Drug Abuse Treatment Clinical Trials Network REFERENCES To explore interventionist characteristics that might account for variance in treatment fidelity observed in the training cases, bivariate correlations were computed between the MITI ratings and selected participant characteristics (See Table 3). Age, counseling experience, and self-reported use and understanding of MI were not significantly related to performance in training cases. Of the SUS subscales, only Psychological Model scores were significantly (and negatively) related to Global Clinical Ratings, MI Spirit, and Percent MI-Adherent. The two-stage interventionist training used in this study produced very good results across the two waves of interventionists trained. Focusing first on fundamentals and later on specific intervention content may bestow an advantage for learning and implementing brief interventions based on a motivational interviewing approach. The significant differences in ratings between the two waves underscore the need for formal fidelity monitoring even when training procedures are highly specified. Effective training in psychosocial treatment modalities is critical to maintaining fidelity in both research and practice. Maintaining fidelity may be particularly challenging in the Emergency Department settings due to the fast pace and competing urgent and emergent priorities. Here we evaluate the effectiveness of a two-stage training for interventionists participating in NIDA Clinical Trials Network Study 0047: Screening, Motivational Assessment, Referral, and Treatment in Emergency Departments (SMART-ED). DISCUSSION TABLE 1: Interventionist characteristics TABLE 3: Regression model of predictors of training scores TABLE 2: Ratings of first two training cases Twenty-two counselors have received training to date across the two waves of the study, plus two booster counselors. Table 1 summarizes counselor demographics and scores on the SUS, as well as non-parametric tests for differences between the wave 1 and wave 2. Wave 1 counselors were older and lower on the Psychological Model scale of the SUS. Wave 1Wave 2 Significance BoosterTotal Female Gender5/89/14N.S.2/216/24 Race AI/AN Asian African-Am. Caucasian Ethnicity Hispanic 1/8 5/8 2/8 0/14 2/14 12/14 1/14 N.S N.S.. 0/2 2/2 0/2 1/24 3/24 19/24 3/24 Age36.0 (9.2)29.9 (3.3) (6.4)37.1 (6.8) Years counseling exp.1.7 (2.3)1.8 (2.5)N.S.5.0 (5.7)2.0 (2.7) Licensed as counselor4/86/14N.S.2/212/24 Understanding of Substance Abuse Disease Model Psychological Model Eclectic Model 11.9 (6.1) 12.3 (2.2) 15.5 (3.0) 9.0 (6.5) 15.3 (2.8) 17.0 (4.0) N.S..021 N.S. 7.0 (0.0) 12.5 (0.7) 20.5 (0.7) 9.8 (6.1) 14.0 (2.9) 16.8 (3.7) Categorical measures shown as proportions, continuous measures shown as Mean (S.D.). Between-group comparisons are for Wave 1 vs. Wave 2, and use Mann-Whitney U for scale variables due to non-normal distributions. Wave 1Wave 2 Significance BoosterTotal Global Clinical Rating MI Spirit < Direction N.S Empathy Reflection to Question Ratio N.S Percent Open Questions N.S Percent Complex Reflections N.S Percent MI-Adherent Global Clinical Rating MI SpiritDirectionEmpathyRatio of Questions to Reflections Percent Open Questions Percent Complex Reflections Percent MI- adherent Age Self-reported use of MI approach Self-reported MI understanding Years of experience Disease Model Psychological Model Eclectic Model MITI Global Clinician Ratings averaged 4.50 ±.56 on a scale of 1-5, well above the threshold for competency (4.0), and higher than scores typically recorded after a two-day training (See Table 2). Scores on the other three summary ratings (MI Spirit, Direction, and Empathy) were also well above the criterion for competency. The four computed scores (Reflection to Question Ratio, Percent Open Questions, Percent Complex Reflections, and Percent MI-Adherent) were all in the “Beginning Proficiency” range, except for Percent Complex Reflections, which was in the Competent range. All but 2 participants met criteria for certification based on the first 2 practice sessions, and all were eventually certified. All interventionists scored 100% on the content checklists for both sessions. Surprisingly, significant differences were found between Waves 1 and 2 on three of the four summary ratings as well as one of the computed scores (Percent MI-Adherent), with higher scores for Wave 1 in each case. TABLE 3: Correlates of treatment fidelity (Pearson’s r, p) Overall, the performance of interventionist on pilot cases in this trial was very good, even though the interventionists were not highly experienced and only half were licensed. This supports the effectiveness of the two-stage training approach that was used. Given small sample size and the large number of comparisons that were made, the inferential findings of this study should be interpreted cautiously. Still, the relatively large differences in performance between waves 1 and 2 are surprising, and are not readily explained by differences between the groups. This raises the possibility that differences in the first (local) phase of the training could have accounted for some of the observed differences. The finding that Psychological Model scores were inversely related to proficiency scores was unexpected, and requires replication. It remains to be seen whether session ratings during the training phase will predict performance during the main trial. Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. M. L., & Miller, W. R. (2005). Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment, 28, 19–26. Humphreys K, Greenbaum MA, et al. (1996). "Reliability, validity, and normative data for a short version of the Understanding of Alcoholism Scale." Psychology of Addictive Behaviors 10(1):